Calibration Gestures For Finger-Mapped Gesture Systems

ABSTRACT

A finger-mapped gesture system is a user interface method and apparatus for rapid data input on a touch-sensitive surface. The user gestures by holding the hand stationary relative to the surface and moving the fingers primarily by extension and flexion. Each finger gestures on a gesture surface instead of selecting among keys. The particular fingers gesturing and the gestures performed indicate the intended user input. On devices that do not directly identify fingers, users first perform a calibration gesture to associate finger identities with regions of the surface. The user may “drum” the fingers generally from one side of the hand to the other, or the user may “squeeze” or expand the hand on the surface with all fingers touching, including the thumb. These calibration gestures free users from having to look at a keyboard or at their fingers to target visual elements while gesturing.

1 CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Divisional of U.S. application Ser. No. 16/037,077filed Jul. 17, 2018, which is a Divisional of U.S. application Ser. No.14/272,736 filed May 8, 2014, now U.S. Pat. No. 10,082,950, whichclaimed benefit of U.S. Provisional Application No. 61/821,224 filed May8, 2013, U.S. Provisional Application No. 61/823,909 filed May 15, 2013,and U.S. Provisional Application No. 61/943,675 filed Feb. 24, 2014, andwhich was a Continuation-in-Part of PCT/US2012/064563 filed Nov. 9,2012, now published as WO 2013/071198, which claimed benefit of U.S.Provisional Application No. 61/557,570 filed Nov. 9, 2011, thedisclosures of all of the above applications being incorporated byreference herein.

2 FIELD OF THE INVENTION

The present invention is a user interface technology. It characterizescomputer devices and processes that receive input from a user. Thetechnology interprets user actions of fingers touching one or moresurfaces as individual user requests. The surfaces can be physical orvirtual, including purely spatial or mathematical surfaces. The actionsof fingers touching surfaces can be direct or indirect, includingfingers only touching surfaces by proxy. The invention is particularlyadapted for inputting characters and text operations and for replacingcomputer keyboards.

3 BACKGROUND ART

Touchscreen devices appear to promise a future of completely portablecomputing, but it has been a challenge to find an effective alternativeto the conventional keyboard. Many portable devices include smallkeyboards, but it is difficult or impossible to touch-type on thesekeyboards, so data entry remains slow and cumbersome. Serious data entrystill requires propping a desktop or laptop on a surface andtouch-typing on a conventional keyboard with both hands. Theconventional keyboard is too big to be portable, and the method forrapid data entry on a conventional keyboard is not conducive to thecircumstances of portability.

The acuteness of the problem has led to a broad variety of methods forinputting into a touchscreen device. The most prevalent is a virtualsoftware keyboard that emulates a physical keyboard in look andfunction. Apple iOS and Android OS devices come standard with virtualkeyboards. Alternative virtual keyboards are also available that arrangethe keyboard keys differently in efforts to improve the rate of userinput. For example, a number of virtual keyboards arrange keys to allowfor faster input using both thumbs at once. MacKenzie & Zhang (1999)describe a virtual keyboard optimized through an analytic analysis ofEnglish. The “Swype” keyboard by Nuance Communications, Inc., also has akeyboard layout, but it does not require that the user lift fingersbetween selecting characters, thus increasing the rate of characterentry. However, input on virtual keyboards remains slow because theentire hand moves when selecting characters and because the user musttake some care to position fingers on the touchscreen for eachcharacter. In recognition of how error-prone it is to require that userscarefully place fingers at specific locations, most virtual keyboardsinclude some form of autocorrection, allowing the user to type at a ratefaster than is required for accuracy. To further improve the rate ofinput, virtual keyboards usually also include a form of autocompletion.Even so, autocorrection and autocompletion themselves introduce errorsrequiring correction, limiting the rate of input.

Numerous approaches have been explored for reducing error rates,reducing interface size requirements, and increasing the rate of input.Consider handwriting recognition systems. Many such systems beenimplemented on touchscreens, including a shorthand system that Goldberg& Richardson (1993) developed to minimize the gesturing required. TheGraffiti system by Palm, Inc., is a famous example of such a system.These systems generally allow users more latitude for error than avirtual keyboard does, but they are no faster than handwriting.Efficient solutions appear to reside with coded input, such as byselecting keys.

Small portable devices must balance providing enough keys to be usefulwith providing keys large enough for fingers to accurately target. Acommon approach to providing character input with few keys is toassociate different characters with different sequences of key presses,as done on twelve-key phones. In US 2002/0136371-A1, Bozorgui-Nasbatimproves on the traditional twelve-key phone system with a system thatonly requires two key presses for every character. Frequently occurringcharacters are input by pressing the same key twice. Less frequentlyoccurring characters are input by pressing a key and an adjacent key.This approach leverages combinatorics to reduce the number of keypresses required, but such series entry solutions require that the usercarefully target keys one after another. Chorded keyboards address thisproblem by assigning simultaneous combinations of key presses todifferent characters. A single pressing of multiple keys inputs aparticular character. However, chorded keyboards require the user tolearn to move fingers independently, as on a piano, and also to memorizea great variety of correspondences between finger combinations andcharacters, making chorded keyboards beyond the skill of most.

The ideal solution appears to reside with combinatorics and coded input,provided that gesturing skill and memorization requirements can beminimized. There are other approaches to inputting characters withcombinations of coded input. One approach is to initiate a gesture on afirst region of the device interface and end the gesture on a secondregion. The combination of the first and second regions then indicatesthe character input. In PCT/US2012/064563, Bonner et al. describe asystem in which the user touches a region of the touchscreen to select aset of characters. The system then presents those characters in a listand allows the user to drag the finger, without first lifting, to asecond region, which corresponds to the character that the user wants toselect, whereupon lifting the finger selects the character. In U.S. Pat.No. 5,521,986, Curtin et al. describes a similar approach, where regionsare selected in combinations to form characters on a simulated LED bylighting appropriate segments. These multi-region selection approacheshave the user carefully positioning both where the finger starts andwhere the finger ends, thus requiring the user to employ feedback fromthe system, such as visual or audible feedback, to ensure that accurateselections are made. Perlin (1998) ameliorates the targeting problem alittle by establishing all of the regions around the position where theuser first places a stylus. The user can then select the first regionsimply by going in the appropriate direction a sufficient distance. Evenso, the user must still carefully target the second region for eachcharacter entered, keeping the user's focus on the user interface whileinputting characters, sacrificing speed for accuracy.

The systems described so far employ combinations of the same class ofinput, that class being the selecting of keys or regions. Morecombinations become available by combining different classes of input.In U.S. Pat. No. 5,341,133, Savoy et al. describe a system of keys thatcan each be pressed not only downward but also in various directions.The key pressed and the direction in which it is moved together select acharacter to input. Direction is therefore another possible class ofinput. Savoy et al. also allows simply pressing a key downward toindicate a character. Twelve keys each supporting four directions ofmovement and direct pressing can together indicate 12*(4+1) or 60distinct characters, each requiring only a single user action, which isthe pressing of a key a particular direction. In U.S. Pat. No.5,521,986, Curtin et al. apply this to regions on a touch-sensitivesurface. They describe the combination of touching a region and moving astylus or finger in a particular direction as indicating a particularcharacter, without requiring the gesture to end in a target region. InU.S. Pat. No. 6,104,317, Panagrossi also describes a user indicating acharacter by touching a region and moving in a direction on atouch-sensitive surface, but further allows the user to simply touch aregion to indicate a character, as done with a conventional keyboard,thus providing an analogue of U.S. Pat. No. 5,341,133 fortouch-sensitive surfaces. In U.S. Pat. No. 7,694,231 B2, Kocienda et al.also describe selecting characters according to whether the user taps orgestures from a region and according to the direction gestured.

Each independent class of input available to a gesture can be thought ofas a parameter of the gesture. The region or key selected is oneparameter, and the direction indicated is another. In U.S. Pat. No.7,057,607 B2, Mayoraz et al. describe several more possible parameters,including how far the finger moves, how fast the finger moves, andwhether the finger goes “back” after moving in a direction. Mayoraz etal. have these parameters complementing parameters for region anddirection in order to select characters for input. With so manyvariations available for each region, fewer regions are needed toprovide the user with direct access to a full character set. Given thespace limitations of portable devices, when there are fewer regions, theregions can be larger, further reducing the probability of user error.However, even with a variety of gesture parameters, all of thecombinatorial systems mentioned so far, other than the chorded keyboard,have the user repeatedly moving a hand as a unit to target a finger to aparticular key or region. This is analogous to fingers typing by “huntand peck” on a conventional keyboard, and it is slow. Users can improvetheir rate of input by developing strategies for where they hover theirfingers and for when a particular finger is used, as people do whenhunting and pecking, but the user can benefit more when the systemitself is specifically designed to accommodate a particular strategy ofinput.

Ideally, the hand would remain stationary as the fingers gesture, asdone when a user touch-types on a conventional keyboard. With the handsstationary, only the fingers need move to gesture, presumably increasingthe rate of input. Several patent applications describe this approach.In US20060077179 A1, Chang et al. describe calibrating a virtualkeyboard according to the maximal extents of a person's hand on atouchscreen device, producing a virtual keyboard on the deviceappropriately sized to the hand, with the keys appropriately spaced. InUS 20090237361 A1, Mosby et al. describe calibrating the positions ofvirtual keyboards according to positions registered for fingers andpalms, with each hand possibly employing its own virtual keyboard. InPCT/NL2011/050911, Vellinga describes a system that infers theidentities of fingers according to their rest locations on atouch-sensitive surface. The system establishes either a user interfacewidget at each finger or a miniature keyboard surrounding each finger.The character indicated is a function of the identity of the widget orkey selected. In US20130113714 A1, Mao describes a similar system thatassigns a keypad to each finger. A finger either taps one of the keysassigned to it or simply moves in a direction of a key to select thekey. Mao also allows for the system to monitor the directions ofmovements of the fingers to calculate “median directional angles” foreach finger. Mao employs the median direction angles to orient eachkeypad to align its keys with the direction in which fingers flex andextend, making the keys easier for the fingers to touch.

Each of these hand-calibrated systems associates individual fingers withmultiple keys, although Mao allows for a finger to indicate a key, andthus a character, simply by the direction of movement of a finger. Maoincludes keyboards that associate some keys each with only one fingerbut suggests using “more keys per finger in order to cover the wholealphabet.” Each of these systems has each finger of a hand movingindependently in any direction. Any of these systems can be conceived insimple form selecting only from two keys along the direction ofextension and flexion for a finger, but none is capable of selectingmore than two characters per finger under this restriction. Threecharacters per finger are possible if tapping a key also inputs acharacter. In order for these systems to provide more than threecharacters to a finger moving only by extension and flexion on astationary hand, multiple keys must be stacked along the finger's nearlylinear path of extension and flexion. If multiple keys are stacked alonga line, the user must accurately place fingers when touching or liftingfrom the surface, in order to select the proper key, forcing the user tocarefully train each finger to target specific locations on the screen.Some users may be able to learn to gesture with such accuracy, butstacking keys probably leaves most users frequently making mistakes.None of these hand-calibrated systems, as described, provides anaccurate and efficient means for rapid character input into atouchscreen.

A solution combining several of the above approaches helps to addresssome of the problems with any individual approach. In particular,extending the Vellinga and Mao combination by allowing a finger to inputmultiple gesture parameters during a gesture gives each finger access tomany more characters without requiring the fingers to target specificregions. For example, the direction that a finger moves and the distancethat a finger moves can together indicate a character. However, existingsolutions that involve gesture parameterization and calibrating for ahand, including Mayoraz et al., have fingers selecting from amongmultiple keys or regions as part of the process of selecting acharacter. Additional insight is needed to realize that individualfingers do not need access to multiple keys or regions, that each fingercan be assigned a single key or region and yet still provide access toall the characters of an alphabet and more. That is, each Vellinga orMao keypad need only have a single region, and the user need only inputgestures with a variety of gesture parameters; the user need not targetindividual fingers to multiple keys.

Suppose a solution combines one of these hand-calibrated systems withMayoraz et al. using only one region per finger. Most of thesehand-calibrated solutions place keys to the left and right of eachfinger, as well as farther from the hand and closer to the hand, andMayoraz et al. has no concept of assigning keys to individual fingers.The combined Mayoraz et al. solution, given the non-obvious constraintof one region per finger, would supplant keys with directions, requiringeach finger to move in any of four directions or more, as with theoriginally described hand-calibrated systems. Mao includes some fingersthat are each associated with only one key, but Mao assigns a letter ofthe alphabet to each of four directions for each key, further suggestingthat the most apparent solution to providing broad coverage ofcharacters at one region per finger requires distributing frequentlyentered characters over four directions.

Moreover, in a combined hand-calibrated Mayoraz et al. solution, if eachfinger is restricted to only moving forward and backward in order tokeep the user from having to realign the hand between characters, anadditional problem arises. Consider the Mayoraz et al. gestureparameters that can be input at high speed: direction, length, andreversing direction. Users can readily be expected to learn to gestureat two gesture lengths, and three gesture lengths may also be possible,though probably with less accuracy. At two gesture lengths, with onlytwo directions and an optional reversal of direction, four fingers caninput 4*2*2*2 or 32 characters. At three gesture lengths the number ofcharacters available is 4*3*2*2 or 48 characters. 32 characters are notsufficient to represent 26 letters and 10 digits. 48 charactersapproaches the number of keys on a keyboard, but it is at a cost ofaccuracy as users attempt to reliably select among three gesture lengthsat high speed. Allowing taps to represent characters increases the countby 4, for four fingers. Mao describes the taps of adjacent fingers alsoindicating characters, but this only adds an additional 3 characters.Hence, it is not clear that restricting fingers to two directions ofmovement is feasible in even a hand calibrated Mayoraz et al. stylesystem, further inclining developers to broadly distribute thecharacters over four or more directions per finger even in this combinedsystem.

Still additional insight is needed. First, it must be understood that auser gestures most rapidly when the hand remains stationary even as thefingers move. Second, an additional gesture parameter is needed thatsatisfies this first constraint and yet still greatly increases thenumber of characters available. Because users do not remembercombinations of fingers well, and because it takes effort to learn touse fingers independently, chorded keyboards are an unlikely source ofinspiration for a solution. Yet chorded keyboards do suggest a solution:the combination of fingers gesturing can further select the characterbeing input. For example, the index finger gesturing in a certaindirection for a certain length would indicate one character, while boththe index finger and the middle finger gesturing for a certain distanceand length would indicate another character. The region that a fingerinitially touches would identify the finger, and multiple fingers couldsimultaneously touch multiple regions and simultaneously perform agesture to input a particular character. The problems associated withchorded keyboards can be avoided by limiting the finger combinationsthat users must learn. Multi-finger gestures are easy to perform whenall of the fingers of the gesture are adjacent, and adjacentmulti-finger gestures can be sorted from left to right for a givennumber of participating fingers, facilitating the association of sets ofgestures with sets of characters.

Prior art includes gesturing from combinations of regions to inputcombinations of characters. In U.S. patent application U.S. Pat. No.8,359,543 B2, Sengupta describes multiple fingers selecting multipleregions and gesturing simultaneously to input a word. Each regioncorresponds to a character, and the combination of characters selectedtogether indicates the word, usually selecting a word that includes thecharacters indicated. There are single character words, such as thewords “a” and “I” in English, but U.S. Pat. No. 8,359,543 B2 appears tobe directed at simplifying the entry of strings of characters, and it isnot clear that the inventor contemplated selecting multiple regions andgesturing to input either single characters or single character words.Even if single character words were within the scope of U.S. Pat. No.8,359,543 B2, it would not be clear that the invention would be able toinput more than just a few characters, because most languages have fewsingle-character words. Moreover, it may not be obvious that anefficient gesture system can actually depend on users gesturing numerouscombinations of fingers for frequently entered characters such asletters, that multiple multi-valued gesture parameters can be reliablydefined for multi-finger gestures, or that multi-finger gestures can bemade more efficient by tailoring the system to the hand.

Finally, there is a fundamental difference between selecting keys andgesturing to input characters that none of these solutions accommodates.When a number of characters are associated with a particular region, theuser selects the region to input one of the characters. If selecting acharacter entails gesturing a distance, as it does when inputtinggesture parameters such as those of Mayoraz et al., the finger may moveoff of its starting region as it gestures. This need not affect thecharacter indicated, but it does inconveniently place the finger for thenext gesture, especially if the next gesture is to travel in theopposite direction. The finger must first locate the intended regionagain before inputting another character. If there is only one regionper finger, this unnecessarily slows the user down and reduces useraccuracy. Regions should instead be sized to accommodate the lengths ofat least the frequently input gestures. That is, the ideal system doesnot provide each finger with a key (or with multiple keys) but ratherwith an area for gesturing.

4 SUMMARY OF THE INVENTION

The present invention is a system for rapid data input via gesturing.This specification refers to the system as a “finger-mapped gesturesystem,” often referencing the system simply as the “gesture system.” Inits basic embodiment, a finger-mapped gesture system is a type of userinterface for inputting data into a touch-sensitive surface. It isanalogous to a virtual keyboard, and it is intended to replace thevirtual keyboard in its most common application. Unlike a virtualkeyboard, the gesture system has no buttons and requires no visual userinterface elements. Instead of identifying inputs according to buttonspressed, a finger-mapped gesture system identifies inputs according tothe particular fingers of a hand that gesture and according to thegestures that the fingers perform. The gestures are designed to allowthe user to input with a hand maintaining a posture similar to theposture of a hand in home position while typing on a conventionalkeyboard. FIGS. 3A-3C illustrate this posture, which is called “fingermapping posture.” Although the gesture system is optimized for textentry, its gestures can be used to indicate any sort of user request,such as requests to perform application functions or to perform musicalnotes. The invention encompasses not only finger-mapped gesture systemsoverall but also various embodiments of these gesture systems andinventive techniques for optimizing their performance.

A finger-mapped gesture system relies on determining the identities ofgesturing fingers, which entails distinguishing the different fingers ofa hand with some reliability. Either the input device provides thefinger identities or the gesture system infers them. When the devicedoes not provide identities, the gesture system divides an input area onthe touchscreen into regions and associates each region with a differentfinger. This collection of regions is called a “finger map.” The regionsare analogous to buttons or keys, but because regions correspond todistinct fingers, there are far fewer of them than there are buttons ina normal character entry system. Moreover, the gesture system does notprovide any one finger with more than one region to choose from,contrary to traditional systems that make multiple buttons available toa finger for inputting characters. The regions of a finger map areideally adjacent and contiguous to optimize the space available to eachfinger. Each region provides a space for its associated finger to eithergesture or touch and hold to perform a shift function. According to asimple embodiment of the invention, the region that a finger initiallytouches when gesturing provides the finger's identity. FIGS. 7A-7Ndepict a variety of possible finger maps.

To employ a finger map, the user positions a hand in finger mappingposture over the input area so that each finger is above its associatedregion. This position is called the “home position” for the hand. Thephysical keys on the home row of a conventional keyboard provide tactilefeedback for properly placing the fingers, but a touch-sensitive surfacenormally cannot provide such feedback. Moreover, optimal gesturing withthis system requires that the fingertips be roughly under the knuckles,preventing the user from spreading the fingers out to use a potentiallylarger one-size-fits-all finger map. The finger map should therefore beconfigured specifically for the particular hand of a particular person.The most important property of a finger map is the relative sizing andspacing of its regions. Regions should be aligned to the natural restlocations of a user's fingers when the user places the hand in fingermapping posture. The gesture system should calibrate the finger map tothese rest locations prior to receiving gestures for user requests.There are various ways to accomplish this, including allowing the userto select from predefined finger maps and allowing the user to variablyselect the spacing of the regions. This specification defines severalcalibration gestures that a gesture system could allow a user to performto specify a hand calibration. FIG. 38 illustrates some of theproperties that may characterize a hand calibration.

Once the user has a finger map suitable for the user's hand, the userpositions the hand over the finger map and gestures, allowing eachfinger to touch its associated region. The primary gestures are“strokes” and “sweeps.” A “stroke” is a gesture in which a single fingertouches the touchscreen, moves in a direction, and then lifts. A “sweep”is a gesture in which two or more fingers touch the touchscreen, move ina direction together for approximately the same distance, and then lift.FIGS. 14A-14H depict fingers in various combinations performing sweepsin various directions. The combination of finger identities used toperform the gesture and the direction in which the gesture movestogether indicate a particular user request. The finger identities andthe direction are gesture parameters of the gesture. The user request isa function of the values of the gesture parameters. Another usefulgesture parameter is gesture level. Gesture level is an indication ofthe distance that the gesturing fingers move, with different ranges ofdistance selecting different gesture levels. For example, a gesturesystem with only two gesture levels has a single threshold distancebelow which one level is indicated and above which the other level isindicated. FIGS. 35A-35D and 36A-36D depict examples of gestures usingthese three gesture parameters to input letters for strokes and sweeps,respectively, where the letters appear in single quotes.

If the hand moves out of home position while gesturing, the hand mustfirst move back to home position to properly align fingers with thefinger map before gesturing again. This is inefficient. To maximize therate of input, the hand should leave home position only infrequently.The gesture system accomplishes this in part by maximizing the number ofuser requests available to fingers that only extend and flex while inhome position. Finger extension and flexion are more intuitively called“forward” and “backward” movement, respectively. Movement along thisdimension is called “longitudinal” movement, and a path available to afinger performing longitudinal movement is called a “longitudinal path.”Restricting each finger to two directions greatly reduces the number ofgestures available to each finger. Multi-finger sweep gestures improvethe situation by allowing users to select fingers combinatorially, butthe chorded keyboard proved that users do not remember arbitrarycombinations very well. Moreover, it is a skill to be able to movefingers independently for arbitrary combinations. Users do readily moveseries of adjacent fingers together as a unit, though, and for a givenfinger count, combinations of adjacent fingers can be ordered on thehand from left to right. For example, the index-middle fingercombination is to the left of the middle-ring finger combination, whichis to the left of the ring-pinky finger combination. These combinationscan be assigned sequential letters of the alphabet, thus minimizing theskill required to perform the gesture while also making thecombinatorial series memorable. The gesture system further improves theefficiency of data input by assigning the most commonly inputcharacters, particularly letters, to longitudinal gestures.

Ideally, one hand would be able to hold a portable touchscreen devicewhile the other performs rapid data entry on the device, particularlyfor inputting characters. This requires making as many characters aspossible available to a single hand that is primarily gesturinglongitudinally. The inventor has found that it is probably best to placecharacters at no more than two distinct gesture levels. Four fingersgesturing in two directions with at most two distinct gesture levels,either singly or in combinations of adjacent fingers, provides access to40 user requests. This is not enough to represent a conventionalkeyboard. It is an option to provide a button that acts as a shift key,but it is more helpful to expand the gesture parameters that areavailable to gesturing fingers. As with direction and gesture level, itmust be possible to input these additional gesture parameters bygesturing each finger along a single longitudinal path while the handremains in home position. Gesture parameters that can select amongmultiple values while restricted to longitudinal paths are“longitudinally selectable” gesture parameters. FIGS. 8A and 8Billustrate the longitudinal selectability of gesture level and gesturedirection for strokes and sweeps, respectively.

It is not sufficient for a gesture parameter to be longitudinallyselectable. The primary purpose of finger-mapped gesture systems is toprovide for the rapid input of data, and rapidly moving fingers cannotreliably target specific areas of the touchscreen during the gesture. Itshould therefore be possible for the user to perform a longitudinalgesture to select the same user request for the same actions of the samefingers, regardless of where the fingers start within the input area ortheir associated regions, and regardless of where the fingers end,subject to boundary constraints that may invalidate the gesture. Such aproperty would also allow the user to remain focused on the data beinginput and not have to look at the fingers to place them. That is, thegesture parameters of longitudinal gestures should be independent oftheir start and end locations except as needed to determine fingeridentities. This property is called “position independence.” It isoverly restrictive to require that gesture parameters beposition-independent, however. There can be utility in treating gesturesthat end outside the input area differently from those that end insidethe input area, provided that the input area is large enough toaccommodate rapid input patterns of gesturing.

Gesture parameters can be further characterized by whether or not theyare “path-dependent.” A “path-dependent” gesture parameter is a functionof a path of a gesturing finger, so that the gesture that the userperforms actually influences the value of the gesture parameter. Eachfigure-mapped gesture includes at least one path-dependent gestureparameter, further distinguishing finger-mapped gesturing from gesturingon chorded keyboards. Gesture level and gesture direction are examplesof path-dependent gesture parameters. “Backbrush level” is anotherpath-dependent gesture parameter. Gestures have a backbrush level ofzero by default, and the backbrush level is increased to one byreversing the direction of the gesture after having moved the fingers ina direction but prior to lifting the fingers. Higher backbrush levelsmay be indicated according to the length of the reversal leg of thegesture, analogously to indicating a gesture level by the length of thefirst leg of the gesture. Gesture level, gesture direction, andbackbrush level can all be implemented longitudinally selectable andposition-independent. For example, FIGS. 8A and 8B illustrate the use ofgesture level and gesture direction to select characters. In thesefigures, it does not matter where any finger begins gesturing within aregion. All that matters are the fingers that gesture, the direction inwhich they gesture, and how far they gesture forward or backward.

This specification defines two gesture parameters that can modifystrictly longitudinal gestures without modifying the gestured paths. Thefirst is the “touch count.” By default, gestures have a touch count ofone because the fingers have only touched the gesture surface once toperform the gesture. To increase the touch count, the fingers involvedin the gesture tap the touch surface one or more times immediately priorto gesturing. To gesture with a touch count of two, the user taps thetouchscreen with the fingers that will be gesturing, lifts the fingers,and then touches again to perform the gesture, all in rapid succession.The user can perform additional taps prior to gesturing to furtherincrease the touch count. The number of touches parameterizes thegesture. The second gesture parameter is the “shift hold.” Being similarto the state of the shift key on a keyboard, a shift hold has a defaultshift mode. A gesture system can readily provide shift buttons forfingers that are not gesturing, such as for the thumb if there is enoughroom on the touchscreen, but the gesturing fingers can also performshifts. A finger that can gesture performs a shift simply by touchingand remaining motionless while other fingers gesture. The combination offinger identities remaining motionless indicates the shift mode andserves as an additional gesture parameter.

Recall that gesturing solutions for inputting characters typicallyensure that multiple keys are conveniently accessible to individualfingers. Were a finger-mapped gesture system to make multiple keysconveniently accessible to each finger, the user might accidently selectone of the keys during a session of rapid gesturing. So while it may beuseful to place keys within reach of gesturing fingers, they are bestplaced at inconvenient distances. Consider that in order to maximize therate at which a user can gesture, it should be possible for the user toperform a gesture in one direction, to briefly lift the finger, and thento place the finger back down on the gesture surface exactly where thefinger just lifted in order to perform a gesture in the oppositedirection. This observation quantifies both a region's ideal size andthe minimum distance at which to place surrounding keys: a region shouldbe at least as long, in the longitudinal dimension, as the longestgesture that the user is likely to perform. Ideally, a region should beas long as the longest available gesture. In a gesture system supportinggesture levels, this is at least as long as a second-level gesture.Recognizing that users tend to begin gesturing near the centers ofminimally sized regions, it is better still for regions to be at leasttwice as long as a second-level gesture. After all, a region is not akey that a finger presses, but rather a surface for gesturing. Theseobservations apply equally well to systems in which the host providesfinger identities and there are no regions, as they still characterizethe ideal minimum length of the input area in the longitudinaldimension.

The specification also describes generalized embodiments offinger-mapped gesture systems. The device can have multiple distinctgesture surfaces on which the user can simultaneously gesture, possiblyone for each finger. Each of these surfaces is called a facet. Fingermaps positioned on the facets can have additional properties that assistthe user with gesturing, beyond just the relative spacing of regions.These properties include shaping the regions to accommodate the naturaldirections of movement of a user's individual fingers. The propertiesalso include associating each finger with a distinct forward andbackward direction, also according to what is most natural for finger.This latter feature is most useful when the gesture system also supportsleft and right finger movement and must accurately distinguish the leftand right directions from the forward and backward directions for eachfinger. Calibration gestures provide convenient ways for users tospecify calibrations for finger maps, enabling multiple users to gesturesimultaneously into different input areas on the same surface. Thegesture surface itself need not be a touchscreen, as finger-mappedgesture systems can be defined for any mathematical surface thatdistinguishes between touching and not touching. The specificationattempts to capture the full scope of the space to which these gesturesystems apply in a generic class of device called a “touch device.”

The present invention is further directed to computer readable mediacarrying executable code for implementing all of the processes disclosedherein, and the present invention is further directed to systems forperforming all of the processes disclosed herein.

5 BRIEF DESCRIPTIONS OF THE DRAWINGS

The drawings reflect the fact that finger-mapped gesture systems canaccommodate gestures that a user inputs either by flexing and extendingthe fingers of a stationary hand or by moving the fingers side-to-sideand thus also moving the hand. Even so, the invention is primarilydirected at gestures that travel paths along which fingers may flex andextend. The paths of extension and flexion are relatively straight incomparison with their lengths, and depicting them as such would havediminished or even masked details and distinctions that are relevant tothe drawings and their accompanying discussion. Most drawings of pathstherefore also represent exaggerations of the possible properties ofpaths of flexion and extension for tutorial purposes. In any case, manyof the illustrated techniques apply regardless of whether the fingersflex or extend.

All measurements and proportions depicted in the drawings are estimatedand not necessarily exact. When symbols are present, the relevantmeasurements, proportions, and positions of elements in the drawings areindicated by the symbols and by relationships among them.

FIG. 1 depicts physical components that may comprise a touch device.

FIG. 2 depicts relationships among the logical components of a touchdevice.

FIGS. 3A-3C depict a hand in “finger mapping posture,” which is thesuggested posture of a hand that is performing finger-mapped input on adevice.

FIGS. 4A-4D depict several input configurations of a two-faceted touchdevice.

FIGS. 5A-5H depict some possible arrangements of input areas into whichusers gesture on facets of a device.

FIG. 6 depicts the role of facets in a multi-faceted touch device andexemplifies mapping multiple facets onto a single virtual gesturesurface.

FIGS. 7A-7N depict a variety of ways to partition an input area into theinput regions of a finger map.

FIG. 8A illustrates the longitudinal selectability of gesture level andgesture direction in single-finger stroke gestures.

FIG. 8B illustrates the longitudinal selectability of gesture level andgesture direction in two-fingered sweep gestures.

FIG. 9 depicts the general process of detecting and interpretingfinger-mapped gestures.

FIG. 10 depicts the process of detecting and interpreting finger-mappedgestures for the case where a hand calibration specifies a finger map.

FIG. 11 depicts a state machine for detecting a gesture.

FIG. 12 depicts an example path of a finger that exceeds a gesturingthreshold by performing an initial slide.

FIG. 13 depicts distance vectors for various points on a path thatexceeds a gesturing threshold.

FIGS. 14A-14H depict a hand performing two-fingered sweeps andthree-fingered sweeps in various directions.

FIGS. 15A-15C depict a hand performing holds with various fingers whilegesturing with the remaining fingers.

FIG. 16 depicts determining the gesture length and sampling angle of atwo-fingered sweep using values computed independently for each finger.

FIG. 17 depicts determining the gesture length and sampling angle of atwo-fingered sweep using values that are intermediate between thefingers.

FIGS. 18A-18H depict a sequence in which the paths of two fingersperform a sweep.

FIG. 19 depicts the general process for resolving finger identitydetermination conflicts.

FIGS. 20A-20B depict two examples of two fingers touching the sameregion with one being reassigned to another region.

FIG. 21 depicts an example of determining the length of a stroke byprojecting its path onto a line.

FIG. 22 depicts an example of determining extent vectors from the pathof a finger.

FIG. 23 depicts multiple paths for two fingers, representing either agesture of just one finger or a gesture with both fingers.

FIGS. 24A-24C are tables exemplifying how the paths of FIG. 23 can beinterpreted as characters.

FIGS. 25A-25D depict several examples of directional partitionings.

FIGS. 26A-26B depict two examples of the process of determining adiscrete direction from a sampling angle and a directional partitioning.

FIGS. 27A-27D depict several ways to define a reference angle for afinger map.

FIGS. 28A-28G depict a process for determining a discrete direction fora two-fingered sweep in which each finger is on a separate facet.

FIGS. 29A-29H depict a sequence in which a finger performs a multi-tapstroke gesture.

FIGS. 30A-30H depict a sequence in which two fingers perform a multi-tapsweep gesture.

FIG. 31 depicts a process for determining user requests when holds aresupported.

FIG. 32 depicts a process for locking holds across multiple gestures.

FIGS. 33A-33H depict fingers performing a variety of hold gestures.

FIGS. 34A-34H depict a sequence in which fingers perform gestures withresidual and locked holds.

FIGS. 35A-35D depict stroke gestures that are position-independent andvector-dependent.

FIGS. 36A-36D depict sweep gestures that are position-independent andvector-dependent.

FIG. 37A depicts example longitudinal paths of a hand on a finger map.

FIG. 37B depicts example longitudinal paths of a hand on an input areaabsent a finger map.

FIG. 38 depicts properties of a channel-based hand calibration.

FIGS. 39A-39B depict longitudinally conformant linear directions.

FIGS. 40A-40B depict longitudinal conformance based on margins.

FIGS. 41A-41F depict justifications for several gesture accommodationlengths.

FIG. 42 depicts an example of a channel-based finger map.

FIGS. 43A-43D depict input regions derived from points acquired via ahand calibration.

FIGS. 44A-44D depict a sequence in which fingers establish finger mapsby tickling.

FIGS. 45A-45B depict ways to identify a finger by means other than theinitial touch point.

FIGS. 46A-46F depict some ways to derive input regions from partial handcalibrations.

FIGS. 47A-47F depict some ways to derive forward angles from partialhand calibrations.

FIG. 48 depicts the general process for receiving drumming gestures tocalibrate an input area for finger-mapped gesturing.

FIGS. 49A-49H depict a sequence for one scenario of establishing afinger map using the drumming calibration gesture.

FIGS. 50A-50H depict a sequence for another scenario of establishing afinger map using the drumming calibration gesture.

FIG. 51 depicts the procedure for receiving a general squeeze gestureand for processing finger-mapped gestures according to properties of thesqueeze gesture.

FIGS. 52A-52B depict hands performing squeeze calibration gestures.

FIGS. 53A-53D depict the determination of a hand calibration from asqueeze calibration gesture.

FIGS. 54A-54C depict evaluations of a stroke and a sweep performed on afinger map that contains a single calibrated input region.

FIGS. 55A-55C depict evaluations of a stroke and a sweep performed on adivided finger map, potentially multiple facets, with all input regionsin one division being calibrated.

FIGS. 56A-56C depict evaluations of a stroke and a sweep performed on afinger map derived from geometric elements, where each finger has itsown forward direction.

FIGS. 57A-57C depict evaluations of a stroke and a sweep on a dividedinput area not having a finger map, where each finger has its ownforward direction.

FIGS. 58A-58C depict a series of gestures input on a divided input areafor which a division contains three input regions spaced via acalibration.

FIGS. 59A-59C depict a series of gestures showing that finger pathsidentical in form can produce different user requests when used incombination.

FIGS. 60A-60B depict a series of strokes and sweeps performed on asingle-faceted device that does not use a finger map but does usegesture direction.

FIGS. 61A-61B each depict a mapping of gesture parameterizations tocharacters for the accompanying sample implementation.

FIGS. 62A-62B depict the state machine of the sample implementation'sfinger tracker component.

FIG. 63 depicts the state machine of the sample implementation's sessionengine component.

FIG. 64 depicts the state machine of the sample implementation's strokedetector component.

FIGS. 65A-65B depict the state machine of the sample implementation'stwo-fingered sweep detector component.

FIGS. 66A-66B depict the state machine of the sample implementation'scalibration detector component.

6 REFERENCES CITED

-   Goldberg, David, & Richardson, Cate. (1993, Apr. 24-29).    Touch-Typing With a Stylus. In Ashlund, S., et al. (Eds.),    Proceedings of the INTERCHI '93 Conference on Human Factors in    Computing systems. Paper presented at INTERCHI 93: ACM INTERCHI    Conference on Human Factors in Computing Systems, Amsterdam, The    Netherlands (pp. 80-87). Amsterdam, The Netherlands: IOS Press.-   MacKenzie, Scott I., & Zhang, Shawn X. (1999, May 15-20). The Design    and Evaluation of a High-Performance Soft Keyboard. In Williams,    Marian G., & Altom, Mark W. (Eds.), Proceedings of the SIGCHI    conference on Human Factors in Computing Systems. Paper presented at    CHI 99: ACM SIGCHI Conference on Human Factors in Computing Systems,    Pittsburgh, Pa. (pp. 25-31). New York, N.Y.: ACM.-   Perlin, Ken. (1998, Nov. 1-4). Quikwriting: Continuous Stylus-based    Text Entry. In Mynatt, Elizabeth, & Jacob, Robert J. K. (Eds.),    Proceedings of the 11th annual ACM symposium on User interface    software and technology. Paper presented at UIST98 Eleventh Annual    Symposium on User Interface Software and Technology, San Francisco,    Calif. (pp. 215-216). New York, N.Y.: ACM.-   U.S. patent application US 2002/0136371-A1, Bozorgui-Nasbat-   PCT/US2012/064563, Bonner et al.-   U.S. Pat. No. 6,104,317, Panagrossi-   U.S. Pat. No. 7,057,607 B2, Mayoraz et al.-   U.S. Pat. No. 5,521,986, Curtin et al.-   U.S. Pat. No. 5,341,133, Savoy et al.-   US20060077179 A1, Chang et al.-   US 20090237361 A1, Mosby et al.-   PCT/NL2011/050911, Vellinga-   U.S. Pat. No. 7,694,231 B2 Kocienda et al.-   US20130113714 A1, Mao-   U.S. Pat. No. 8,359,543 B2, Sengupta-   http://en.wikipedia.org/wiki/Letter_frequency (Aug. 28, 2013)

7 DETAILED DESCRIPTION OF THE INVENTION 7.1 System Architecture

A user inputs a finger-mapped gesture into a “touch device” thatincludes one or more surfaces that the user touches. Each of thesesurfaces is called a “facet.” The facets may support multiple hands orthey may operate together in a way that supports input across them by asingle hand. It is sufficient for a touch device to have a single faceton which multiple fingers of a single hand may gesture. To simplifyexposition, this specification maps the surfaces of all of the facets toa single mathematical surface called a “gesture surface,” so that eachfacet occupies a portion of the gesture surface. One or more areas ofthe gesture surface—each encompassing portions of one or more facets—aredesignated for receiving finger-mapped gestures. Each of these areas iscalled an “input area” 8. A class of program called a “gesture system” 7monitors each input area for gestures and translates the gestures into“users requests,” which it sends to an external host system to act on.

7.1.1 Basic Terminology

A “gesture input device” is a hardware device that is capable ofreceiving gesture input from fingers. A “finger” is any finger of auser's hand, including the thumb. An “action” or “user action” is anactivity that a user performs with one or more fingers while inputting agesture into the device, such as touching the device at a certainlocation or moving a finger while touching the device. A user inputs agesture to cause the device to produce an effect that is specific to thegesture.

A user gestures into a “gesture interface,” which is a configuration ofhardware and program logic that is capable of receiving gestures. Agesture input device may correspond to a single gesture interface, but asingle input device may also support multiple logically distinct gestureinterfaces. For example, multiple areas of a single touchscreen can eachbe a distinct gesture interface capable of receiving gestures. Thisallows different areas of the touchscreen to support different kinds ofgestures, and it allows the meaning of any kind gesture to be a functionof the area into which the gesture is input. It also allows thetouchscreen to independently input from multiple hands at once.

Each gesture interface employs a gesture system. The “gesture system” 7is the program logic that attempts to interpret finger actions asgestures. The gesture input device reports finger actions to the gesturesystem in the form of “input events.” The device need not pass inputevents directly to the gesture system, as it could first report fingeraction information to an intervening operating system or application,which then packages the information in the form of input events andpasses the input events to the gesture system. The gesture systemprocesses individual series of input events, attempts to recognizespecific types of gestures, and reports the results of the attempts asgesture outcomes. The gesture system need not report unrecognizedgestures.

A “gesture outcome” is a characterization of an attempted gesture. Eachis either a user request or an indication that no gesture wasrecognized. A “user request” is a specification for a behavior,particularly a specification indicating a behavior that a user appearsto be asking a host to perform. A “behavior” is a specific function thatthe host can perform, such as inserting a character into text, deletinga character from text, moving the cursor, or invoking a menu item. Forexample, a user may input a gesture whose gesture outcome is a userrequest to insert text from the operating system clipboard. A userrequest either directly identifies the behavior, in which case thegesture interface has knowledge of the behaviors available, or itindirectly identifies the behavior via a characterization of thegesture, in which case the host takes responsibility for determining theintended behavior. For example, a user request might characterize agesture as a long forward stroke of the index finger, which the hostmight interpret as a request to insert the letter ‘f’.

The “host” 4 is the system external to the gesture system with which thegesture system communicates. The host includes the operating system orapplication to which a gesture system reports gesture outcomes and thatresponds to the gesture outcomes by performing behaviors. The hostnormally provides the gesture system with input events. However, inimplementations where the device messages the gesture system directly,the host may also include the gesture input device. This general use ofthe word “host” helps to reduce the permutations of concern to thisspecification.

The specification only concerns itself with the kind of gesture inputdevice that is capable of supporting finger-mapped gestures. This classof device is called a “touch device.” Henceforth, all references to a“device” are references to a touch device. In addition, unless otherwisestated, a reference to a “gesture system” is a reference to afinger-mapped gesture system.

7.1.2 Touch Devices and Facets

A “touch device” is a gesture input device that receives input via oneor more surfaces that users touch with their fingers. Each surface iscalled a “facet” of the device. Facets include touchpads that do notserve as displays, such as those included with laptop computers, andtouchscreens that also serve as displays, such as those included withsmartphones and tablets. Some emerging technologies detect touch onarbitrary surfaces, allowing nearly any surface to serve as a facet. Thesurface may even be curved if the curve is gentle enough to supportfinger-mapped gestures over short distances. The device need not requirethat fingers touch facets directly, as the device can instead monitorphysical extensions of fingers or other proxies for fingers.

What it means for a finger to “touch” a facet depends on the specifickind of touch device. Examples of possible kinds of touching include thefinger simply making contact with a facet, the finger pressing with acertain degree of pressure, or the finger passing through a facet thatis permeable or that is only spatial or virtual in construct. Mechanismsfor detecting touch include resistance sensing, capacitance sensing, andmonitoring fingers by camera, laser, or even sonar. Ideally, the deviceprovides the user with feedback indicating when a finger is touching afacet in order to help users become effectively at gesturing. A user'ssense of touch provides this feedback when touching tangible surfaces.

The touch device must allow for input from at least two fingers of thesame hand, though it is not required that the fingers be simultaneouslytouching. That is, it must be possible for a user to hold a handstationary and touch either of two fingers to a facet of the device orto multiple facets of the device. The device must register a distinctionbetween the fingers, such as by directly identifying that the fingersare different or by determining different positions on the facets forfingers touching at different positions. Distinguishing three or morefingers may require heuristic techniques.

A touch device reports positions for fingers that are touching or havetouched facets. These reports indicate the facet touched, if necessary,and where on the facet the finger is touching. The device may alsoreport the positions of fingers that are not touching facets, but forpurposes of this specification, only reports of the positions oftouching fingers are considered input events. To support finger-mappedgestures, a touch device need only report the position at which a fingerinitially touches the facet and the position at which the finger finallylifts from the facet. If the touch device reports the changing positionsof a finger on a facet as the finger moves while touching the facet, thedevice need only report a sampling of the changing positions and notnecessarily every detectable change of position, depending on the rateat which the device reports positions.

7.1.3 Host Context

A touch device that supports finger-mapped gesturing can be used as areplacement for a keyboard, a keypad, a touchpad, or a touchscreen. Anycontext suitable for using these devices is potentially suitable forusing a finger-mapping touch device. This includes using touch devicesto control computers, tablets, mobile phones, navigation systems,computerized office equipment, security access systems, and consumerelectronics in general.

A finger-mapping touch device is also useful in ways that conventionalkeyboards are not. This is a consequence of allowing the user to enter agreat variety of values on a small area using only one hand withouthaving to look at the hand. Examples of such alternative uses includegesturing to control games, to perform taxonomic dissections, to performsurgery, or to control machines in real-time, such as industrialmachines or mobile robots that require the user's attention be somewhereother than on the user's hands. Touch devices may even be useful in acar for allowing people to interact with complex automobile systemswithout having to take their eyes off of the road.

This invention was created in part to enable the rapid input of textunder a variety of physical circumstances, enabling new levels ofconvenience for data input. For example, touch devices could be createdto allow gesturing on arbitrary surfaces such as doors, safes, anddining tables. Perhaps the ultimate convenience in data entry would takethe form of flexible touch pads woven into clothing to enablefinger-mapped gesturing anywhere and at any time.

FIG. 1 depicts physical components, not necessarily distinct, that existin the context of a touch device. The user interface device 1 is ahardware component that is capable of receiving finger-mapped gesturesfrom a user. The processor/controller 2 is a component that receivesuser requests from the user interface device 1. The user requests arecontrol signals or commands. The processor/controller 2 interprets theuser requests for the controlled apparatus 3, translating them into thehardware-specific device protocol sequences that the controlledapparatus 3 is capable of interpreting. In response to the translationsthat the processor/controller 2 provides, the controlled apparatus 3performs behaviors that interact with the physical world. Finger-mappedgestures input into the user interface 1 device therefore inducephysical behavior in the controlled apparatus 3.

A touch device can be a user interface device 1, it can be a device thatcombines a user interface 1 with the processor/controller 2, or it canbe a device that combines a user interface 1 with a processor/controller2 and a controlled apparatus 3. Consider a few example scenarios. In thefirst scenario, a touch device communicates with a desktop computer,which sends radio wave control messages to drive a remote-controlledcar. In the second scenario, a tablet computer operating as a touchdevice sends radio wave control messages to drive a remote controlledcar. In the third scenario, the touch device is the car, and the carcontains a built-in user interface device that directly issues controlinstructions to drive the car via device interfaces on the car.

7.1.4 The Gesture Surface

This specification associates each touch device with a “gesturesurface.” The gesture surface is a two-dimensional surface representingpositions on the device's facets. Every position on the gesture surfaceis given by a coordinate pair, also simply called a “coordinate.” Eachposition on a facet corresponds to a coordinate on the gesture surface,so that each facet is represented by a distinct area of the gesturesurface, with the areas of no two facets overlapping. The gesturesurface is purely a mathematical construct, so the extent of the gesturesurface exceeds facet boundaries, encompassing the full range ofcoordinates available to the two-dimensional coordinate system. Thisallows the intermediate values of mathematical computations to representvalid positions on the surface; not all positions on the gesture surfaceneed represent facet positions that a finger can touch.

The gesture surface is an imaginary construct, a tool for simplifyingthe language of this specification. It need not have representationwithin a touch device. This specification defines a mapping betweenfacets and the gesture surface so that a reader of this specificationcan apply discussion expressed in terms of the gesture surface to thefacet or facets underlying the gesture surface. In this way, forexample, the specification can refer to an area of the gesture surfacethat encompasses a single facet on one touch device and multiple facetson another touch device and yet remain valid for both touch devices,because by referring to that area the specification is actuallyreferring to whatever facets are in the area. As another example,consider that one device may uniquely associate a particular finger witha particular facet, while another device may identify a particularfinger according to its position on a facet. The coordinate of thefinger on the gesture surface is sufficient to identify the particularfinger in both cases, because the gesture surface coordinate is uniqueto a facet; in both cases, the identity of the finger is simply afunction of a gesture surface coordinate.

The mapping of facets to gesture surface is defined as follows. If thetouch device has only one facet, the gesture surface contains thecoordinate system of the facet, so that positions on the facet are alsocoordinates on the gesture surface. If multiple facets compose the touchdevice, either the touch device translates facet positions to a commonfacet-independent coordinate system before reporting them or it reportsboth the facet and a facet-specific position. If the touch devicetranslates facet positions to a common coordinate system, the gesturesurface simply contains this common coordinate system, as it would for asingle-facet device. If the touch device reports facets andfacet-specific positions, each facet maps to a distinct bounded area ofthe gesture surface, with no two bounded areas overlapping. Thelocations and orientations of the areas to which different facets map onthe gesture surface can be arbitrary, because finger-mapped gestures canidentify fingers by region of the gesture surface, and because eachregion can have its own indication for the forward direction of fingermovement. Regardless of the number of facets, if any of the facets arecurved surfaces, it may be necessary for the gesture surface to also becurved, so that the gesture system measures distances appropriately.Just as a two-dimensional mathematical surface need not be planar, thegesture surface need not be planar. For example, the surface of a sphereor a cylinder is two-dimensional and yet curved.

As part of the language simplification, the specification refers tofingers “touching” the gesture surface. Specifically, a finger isconsidered to be touching the gesture surface at a coordinate exactlywhen the finger is touching a facet—and a position on the facet—thatmaps to the coordinate. In addition, a gesture surface coordinateassociated with an input event is considered to be part of the inputevent for purposes of reading the specification. This is onlysignificant for multi-faceted devices that do not map positions ondifferent facets to a common coordinate system before reporting them.For such devices, any reference in this specification to a gesturesystem coordinate is to be interpreted as a reference to the facet andfacet-specific position that maps to the coordinate. This equivalencedefinition allows the specification to be expressed in terms of gesturesystem coordinates and ignore facet-specific positions. Within thecontext of this specification, locations detected for a finger arelocations of the gesture surface that the finger has touched.

Consider a few examples. FIG. 6 depicts a touch device that constrainseach finger to a separate lane 51-54 in order to prevent fingers fromaccidentally touching unintended regions. The device also puts each lane51-54 at a different vertical level, particularly for increasing thecomfort of the index and pinky fingers. This configuration allows thehand 60 to gesture more easily with the wrist bent. Each lane is aseparate facet 51-54, and each facet maps to a distinct region 55-58 ofthe gesture surface 59. The device may report coordinate pairs for theposition of a finger on a facet, but it need not. It could insteadreport a single number indicating the position of the finger along thelength of the facet, in addition to indicating the facet. In this lattercase, the facet positions map to non-overlapping line segments on thegesture surface 59. Each facet corresponds to a different finger, soindicating the facet also identifies the finger. However, a finger canalso be identified given its coordinate on the gesture surface, becausethe coordinate occurs in one of the regions or line segments to whichthe facet positions map and that region or line segment is specific tothe same finger as its corresponding facet. For example, in FIG. 6, acoordinate 61 in region 55 corresponds to a position 62 in facet 51 andhence to the index finger of the hand 60.

Also consider the examples in FIGS. 4A-4D. These figures together depictfour devices 15-18. Each of these devices 15-18 has two surfaces 23-30delimited by a physical barrier 19-22. The barrier 19-22 serves as aguide that both keeps the fingers from crossing to the other surface andhelps the fingers immediately adjacent to the barrier 19-22 track alonga particular line. Each of these devices 15-18 can be constructed with asingle facet that just happens to have a physical barrier 19-22 lying ontop of it. The devices depicted 15-18 can even all be the same kind ofdevice with a mobile barrier shown in different positions. In eitherscenario, the device reports all fingers in the same coordinate systemand need not report the facet. However, each of these devices 15-18 mayalso be constructed with two physically distinct facets. In this lattercase, the device could employ a common coordinate system across thefacets and treat them as if they belonged to the same surface.Alternatively, the device could treat the facets as distinct surfaces23-30 with positions that must be reported alongside an indication forthe facet. A facet that is only capable of receiving input from aparticular finger would be treated as previously described for facets inFIG. 6. However, if some of the facets are capable of receiving inputfrom multiple different fingers, the gesture system can identify thesefingers partly as a function of their particular coordinates.

7.1.5 Input Areas And Sessions

The host manages the gesture surface and may assign different purposesto different areas of the gesture surface. Remember that portions of thegesture surface correspond to facets, so this is equivalent to the hostassigning different purposes to different areas spanning all or parts ofthe facets. For example, the host may designate a portion of the gesturesurface for gesture input and reserve the remaining portion for displayonly, indicating that some portions of facets are to receive gesturesand some portions of facets capable of display are to be relegatedexclusively to display. As another example, the host may designatedistinct areas of the gesture surface for input by different hands,designating one or more facets or portions of different facets each to adifferent hand. Each area of the gesture surface that the host makesavailable to a finger-mapped gesture system is called a “surfaceallocation.” The gesture system employs either all of the surfaceallocation or a portion of it for gesture input, possibly leaving theremainder for use by the host. The portion of a surface allocation thatthe gesture system employs is called an “input area” 8. Surfaceallocations and input areas only ever include coordinates thatcorrespond to facet positions.

FIGS. 5A-5H illustrate some possible arrangements of input areas. Theinput areas 32-42 are the shaded portions of the figures. FIGS. 5A-5Feach show a single-facet rectangular touchscreen 49. In FIG. 5A, theentire touchscreen 49 is used as an input area 32, allowing the gesturesystem to detect finger-mapped gestures no matter where they areperformed. FIGS. 5B, 5C, and 5F each depict a single input area 33, 34,or 39. FIG. 5B depicts an arrangement that is suitable for atouch-sensitive phone, where the input area 33 replaces the virtualkeyboard. The arrangement in FIG. 5C might be suitable for a tablet inlandscape, allowing the host to display information to the left of theinput area 34. FIG. 5F illustrates how the input area 39 can have anyorientation on the touchscreen 49. The arrangements in FIGS. 5D and 5Eeach provide two input areas 35-38. The arrangement in FIG. 5D issuitable for multiple users providing simultaneous input, such as whileplaying a game. The arrangement in FIG. 5E is suitable for separate,simultaneous input by the left and right hands. Because input areas neednot display visual elements, the entire touchscreen 49 is available tothe host for rendering. However, it can be helpful to display an inputarea 32-42 and provide visual support for gesturing.

The boundaries of an input area are not constrained by facet boundaries;a given input area can reside entirely within the gesture surface areadesignated for a single facet, or it can span multiple facets, or it canpartially span facets. This allows a multi-faceted device to supportmultiple input areas, whether simultaneously or not. FIGS. 5G and 5Heach show a touch device having three facets 43-48. In the arrangementof FIG. 5G, the facets 43-45 are all part of the same input area 40,which spans an area of the gesture surface. One hand gestures on allthree facets 43-45. In the arrangement of FIG. 5H, the gesture surfacecontains two input areas 41-42, one for each hand, with each input area41-42 containing a portion of the central facet 47. This latterarrangement allows the left and right hands to gesture simultaneously,each on its own input area 41-42.

Each input area is monitored by a separate instance of a finger-mappedgesture system. An instance of a gesture system is called a “gesturesession” or just a “session.” A gesture system is the logic for a classof gestures, while a gesture session is an instance of that logicrunning for a particular input area. The distinction is not ofteninformative but is relevant here. An input area, its gesture session,and its associated device facets together represent a finger-mappedgesture interface. The terms “gesture system” and “gesture session” areinterchangeable except when it is important to distinguish amongmultiple concurrent sessions or to refer to the changing state of thesession.

The host may dynamically add and remove input areas. The host does thisby creating gesture sessions on surface allocations and by subsequentlydestroying those gesture sessions. For example, touchscreen devicestypically pop up a virtual keyboard on a rectangular surface allocation.Instead of establishing a virtual keyboard, the host can establish aninput area for finger-mapped gesturing on the surface allocation. Thehost does this by creating a gesture session and providing the gesturesession with the surface allocation. The gesture system employs all orpart of that surface allocation as an input area for receivingfinger-mapped gestures. As another example, the host may allow the userto dynamically place input areas on the gesture surface and remove themwhen done. In this case, the host receives user input for positioning aninput area on the gesture surface and in response provides a surfaceallocation at that position to a gesture system, which then establishesan input area on that surface allocation. Multiple users may even usethe same gesture surface simultaneously, each dynamically adding inputareas as desired.

The host may provide special gestures for adding, relocating, andremoving input areas. The gestures for adding and relocating input areasare called “calibration gestures.” The system that monitors for thesegestures is called the “surface monitor” 5. The surface monitorinitially monitors the gesture surface for a calibration gesture. Uponreceiving that gesture, the host—perhaps by way of the surfacemonitor—establishes a gesture system for a surface allocation at thegesture's position, where the gesture system then establishes an inputarea. Once one or more input areas are present on the gesture surface, ahost may even subsequently allow a calibration gesture to be performedon top of an existing input area, provided that the calibration gesturecan be disambiguated from finger-mapped gestures. To accomplish thedisambiguation, the host sends input events to both the surface monitorsession and the finger-mapped gesture session. The first of thesesessions to recognize and report a user request wins, and the host actson that user request. The host also asks the losing session to reset sothat it can attempt to match subsequent gestures anew. When the hostprovides only one input area, the finger-mapped gesture system can alsobe the surface monitor.

An input area is an interface element. The convention for touchscreendevices is that an interface element only responds to input events forfingers that originally touched the touchscreen within the interfaceelement. This specification assumes this convention for input areas on agesture surface: a gesture session only responds to input events fromfingers that originally touched the gesture surface in the gesturesession's associated input area. The host may arrange for this by onlyhanding a gesture session events that originate in the gesture session'sinput area, or the host may hand a gesture session all events thatoriginate in the gesture session's surface allocation and leave it tothe gesture session to ignore input events originating outside the inputarea. This constraint makes each finger-mapped gesture interface anisolated system. Henceforth, the discussion of finger-mapped gesturestherefore unambiguously refers to a single input area, a single gesturesystem, and a single gesture session. Moreover, when a finger initiallytouches the gesture surface, it is initially touching the single inputarea. The finger need not subsequently remain in the input area,however.

7.1.6 Component Relationships

The host 4, input area 8, and gesture system 7 are logical components ofa system that implements finger-mapped gesturing. Some systems may alsoinclude a surface monitor component 5. FIG. 2 illustrates relationshipsamong these components. The graphic notation borrows from UnifiedModeling Language (UML) but is not strictly UML. Instead of defining therelationships, this figure strives to depict the relationships describedelsewhere in this specification for these components. The components canbe implemented as hardware, as software, or as a combination of hardwareand software. The input area 8 can even be implemented as a purelymathematical entity within hardware or software. This specificationtreats the input area 8 as a mathematical entity. It is common practicein object oriented programming to encapsulate both hardware andmathematical entities in software objects, so the treatment of the inputarea 8 as a mathematical entity should not be construed as limiting theimplementation of the input area 8.

A host 4 that allows calibration gestures to place an input area 8 doesso via a surface monitor 5, which establishes each input area 8 and eachgesture system 7 that monitors each input area 8. FIG. 2 depicts thisrelationship with dashed arrows 9-10 pointing to the components that thesurface monitor 5 establishes. Once established, the host 4 sends inputevents to each gesture system 7, and the gesture system 7 sends userrequests and other status information back to the host 4. The input areacomponent 8 can be part of the host 4, part of the gesture system 7, ora distinct entity. If it is represented mathematically as part of eitherof these other components, the input area component 8 will not itselfsend or receive information. For example, the host 4 could filter eventsthat it sends to the gesture system 7 based on whether it deems theevents to be relevant to the input area 8 of the gesture system 7, orthe host 4 could send all events to the gesture system 7 and let thegesture system 7 filter them by relevance to the input area 8 that itmonitors. However, it is also possible for the input area 8 to be adistinct hardware component and thus preclude the need to filter eventsfor the input area 8 because it would receive the events directly.Because it is possible for the input area 8 to be a non-communicatingmathematical entity, no arrows are shown on the dotted lines 11-12 thatassociate the input area 8 with either the host 4 or the gesture system7.

When there is no surface monitor 5, there is one input area 8 and onegesture system 7, although the input area 8 and gesture system 7 mayrequire activation. A gesture system 7 can be implemented to acceptcalibration gestures to recalibrate an input area 8, without employing asurface monitor 5. That is, the surface monitor 5 and the gesture system7 may each behave as a component that receives and applies calibratinggestures. As explained later, this general class of component is calleda “calibrating system” 6. FIG. 2 depicts this relationship by showing asurface monitor 5 and a gesture system 7 as each being a type ofcalibrating system 6.

7.2 Gesture Taxonomy

There are many terms for characterizing gestures. Some are general termsrequiring precise definition, some are basic geometric terms, some areterms that apply generically to gesturing, and some are specific tofinger-mapped gestures. This section defines these terms for thediscussion of finger-mapped gestures in later sections.

7.2.1 General Definitions

Most of the following terms have generally understood meanings, but theyare defined here according to the meanings they have in thisspecification:

-   -   collection—A “collection” is an unordered group of things that        can contain the same thing more than once. For example, the        gesture system may collect numbers into a collection for the        purpose of later averaging the numbers together. If it collects        the same number more than once, the number must occur more than        once in the collection in order to produce a proper average.    -   set—A “set” is an unordered group of things that never contains        more than one of the same thing. It is a collection with no        duplicates. A set of numbers never contains the same number more        than once, for example. A range of angles is an example of a set        of numbers, because there is no reason for an angle to be listed        more than once in a range.    -   subset—A “subset” of a set S is a set of things each of which is        a member of S. A subset of S may be identical to S. A “proper        subset” of a set S is a subset of S that is not identical to S.        These are the mathematical definitions of “subset” and “proper        subset.”    -   series—A “series” is an ordered group of things that can contain        the same thing more than once. It is like a collection except        that the members are ordered. Sequential members of a series are        “adjacent” in the series. The first and last members of a series        are called the “end-of-series” members of the series. The        members of a series S that is a subset of another series T are        ordered in S according to their order in T unless stated        otherwise. A series may also be called an “ordering” to        emphasize the fact that its members are ordered.    -   partially ordered set—A “partially ordered set” is a group of        things some of whose members are ordered relative to one another        and some of whose members are not. The term has the same meaning        in this specification that it does in formal mathematics.    -   distinct—Given two collections A and B, each member of A is        associated with a “distinct” member of B if, and only if, each        member of A is associated with a member of B and no two members        of A are associated with the same member of B. The term also        applies to the members of a collection. The members of a        collection are “distinct” exactly when no two members of the        collection are identical. In addition, given entities X and Y, X        is “distinct from” Y if, and only if, X and Y are not the same        entity.    -   unique—Given two collections A and B, each member A is “unique”        in B if, and only if, every member of A is a member of B and no        two members of B are identical to the same member of A. X is        “unique among” a collection A if, and only if, X is a member of        A and no other member of A is identical to X.    -   size—The “size” of a group of things is the number of things in        the group.

7.2.2 Geometric Definitions

Gestures are geometric notions, and a discussion of them requiresagreeing on the definitions of terms that describe them. This sectiondefines a number of geometric terms according to their use in thisspecification.

-   -   location—A “location” is a coordinate pair in the coordinate        system of the gesture surface, given by coordinate values (x,        y). In this specification, a location is different from a        “point,” which includes a parameter of time. A location on the        gesture surface is said to be on a particular facet if, and only        if, a position on that facet maps to the location on the gesture        surface.    -   average location—The “average location” of a set of locations is        a location having an x-coordinate value equal to the average of        the x-coordinate values of the locations of the set and having a        y-coordinate value equal to the average of the y-coordinate        values of the locations of the set.    -   geometric element—A “geometric element” is a set, either finite        or infinite, of one or more locations. Examples of geometric        elements include locations, lines, line segments, curves,        polygons, and circles, as well as combinations of these. Each        location of a geometric element is said to be “in” the geometric        element or “on” the geometric element. Each geometric element        also “contains” and “has” each of its locations. According to        this definition, a geometric element is an abstract structure        that is independent of how it is created or represented.    -   form—The “form” or “geometric form” of a geometric element is a        characterization of the shape, orientation, and size of the        geometric element. Two geometric elements that differ in any of        these properties have different forms.    -   origin—The specification employs the term “origin” in several        contexts, but the “origin” of a coordinate plane is the location        (0, 0).    -   line—A “line” is normally a line in the geometric sense. In        particular, a line is straight and infinitely long within the        dimensions of the gesture surface that contains the line. A line        on a Euclidean plane is intuitively “straight,” while a        mathematically “straight” line on a non-Euclidean surface, such        as a sphere, is curved from the perspective of Euclidean        geometry. However, curves and line segments depicted in drawings        are also called “lines.”    -   line segment—A “line segment” is the portion of a line between        two different locations on the line and includes both locations.        Each of these two bounding locations is called a “terminus” of        the line segment. (The plural of “terminus” is “termini.”)    -   vector—A “vector” is a line segment in which one terminus is        called the “starting location” and the other the “ending        location.” A vector “starts” at its starting location and “ends”        at its ending location. A vector is said to “point” from its        starting location to its ending location. It is also said to        “originate” at its starting location and to “terminate” at its        ending location. The “length” of a vector is the distance        between the vector's starting and ending locations.    -   angle range—An “angle range” is a range of angles between two        delimiting angles. A range that includes a delimiting angle is        said to be “inclusive” of that angle, and a range that excludes        a delimiting angle is said to be “exclusive” of that angle.        Angles are given in radians, unless stated otherwise, and angle        ranges are expressed using mathematical interval notation. The        notation provides the two delimiting angles and for each        delimiting angle indicates whether the delimiting angle itself        is included within the range. The notation combinations are as        follows:        -   [a, b]—the range includes both delimiting angles a and b.        -   (a, b)—the range excludes both delimiting angles a and b.        -   [a, b)—the range includes angle a but excludes angle b.        -   (a, b]—the range excludes angle a but includes angle b.    -   direction angle—A “direction angle” is an angle that represents        a direction on the gesture surface. This specification reports        direction angles in radians in the range (−π, π], unless        otherwise specified. Each angle is relative to the coordinate        system of the surface, as reported in input events sent to the        gesture system. By convention, a direction angle of 0 indicates        the direction of increasing x-coordinate value for an unchanging        y-coordinate value. This is the direction of the positive        x-axis. Angle values increase counterclockwise relative to this        direction and decrease clockwise. For example, if the user        orients the hand relative to the gesture surface so that the        x-value increases from left to right, an angle of 0 is directly        right and an angle of it is directly left. If in this same        orientation y-values increase from the bottom to the top of the        surface, an angle of π/2 points directly to the top of the        surface, and an angle of −π/2 points directly to the bottom of        the surface. Any angle can serve as a direction angle by        expressing it as its equivalent in the range (−π, π].    -   class equivalence—Two angles are “class equivalent” if, and only        if, they differ by an integer multiple of 2π. The difference        between two angles Q and R “by class equivalence” is the        smallest difference possible between class equivalent angles of        Q and R. This difference is also the “class-equivalent        difference.” For example, the angles 5π/4 and −3π/4 are class        equivalent, the angles 3π/4 and −3π/4 differ by π/2 by class        equivalence, and the class-equivalent difference between the        angles 0 and 4π is 0. Given an arbitrary angle, the angle is        equivalent to a single direction angle by class equivalence;        class equivalent angles represent the same direction.        Comparisons by class equivalence allow direction angles near π        and −π to be compared as if they were near each other. For        example, the following pairs of angles all differ by less than        π/6 by class equivalence: π and 8π/7; it and −6π/7; −π and 6π/7;        and −π and 20π/7. Angles that are near in the directions that        they represent are near in their difference by class        equivalence.    -   length of arc—The “length of arc” that a set of angle spans is        the largest class equivalent difference between different of        angles of the set. This is also the “arc length” of the set of        angles. The set of angles is said to “span an arc” of a given        degree or radian arc length.    -   nearest bisection—The “nearest bisection” (or “nearest bisecting        angle”) of two direction angles Q and R is the direction angle        that bisects two class equivalent angles of Q and R that differ        by less than π. For example, the nearest bisection of the angles        π/2 and π is 3π/4, which is also their normal bisecting angle.        However, the nearest bisection of the angles 3π/4 and −3π/4 is        it because 3π/4 and 5π/4 are class equivalent angles that both        differ by less than it and have a bisection that is also a        direction angle. Note that the angle 0 also bisects 3π/4 and        −3π/4, but 0 is not the nearest bisecting angle. The nearest        bisection of a range of angles is the nearest bisection of the        angles that delimit the range. The nearest bisection is not        defined for direction angles that differ by exactly π, so a        gesture system may either choose it arbitrarily or fail to        recognize a gesture requiring the nearest bisection.    -   nearest average—The “nearest average” (or “nearest average        angle”) of a collection of direction angles is an average of        class equivalent angles for the angles in the collection, where        the class equivalent angles are selected so that both (a) their        average is a direction angle and (b) the largest and smallest of        them differ by less than π. The nearest average is not defined        when such a collection of class equivalent angles cannot be        found. A “weighted nearest average” of a collection of direction        angles is a weighted average of the class equivalent angles that        meet these criteria; the “weights” for each angle are multiplied        by the class equivalent found for the angle. Note that the        nearest bisection of two angles is also the nearest average of        the two angles.    -   vector angle—The “angle of a vector” with starting location (x1,        y1) and ending location (x2, y2) is the direction angle given by        the standard a tan 2( ) function as follows: a tan 2(y2−y1,        x2−x1). Roughly speaking, this is the angle between the vector        and the coordinate system's positive x-axis after the vector has        been geometrically translated to originate at the coordinate        system's origin.    -   line angle—A “line angle” or “angle of a line” is an angle of a        line equal to either of the possible angles that a vector has        when it is coincident with the line. The two possible angles        differ by π radians (180 degrees). Either of these angles,        combined with any location on the line, is sufficient to fully        specify the line. In particular, any location and angle specify        a line. The orientation of a line can be given by a line angle        for the line. The orientation of a line segment can be given by        a line angle for a line that coincides with the line segment.    -   linear direction—A “linear direction” is an indication of two        opposite directions. The term is an abstraction for the        directionality or orientation of a line. Like a linear        direction, every line indicates two opposite directions, but        unlike a linear direction, a line also has location. The slope        of a line indicates two opposite directions independently of        location and can therefore represent a linear direction. A line        also has one line angle for each of the two directions along the        line, and each line angle can be derived from the other by        adding or subtracting π radians (180 degrees). An angle is        therefore also sufficient to represent a linear direction, with        every two opposite angles representing the same linear        direction. To contrast linear directions with ordinary        directions, consider that each possible ray that originates at a        location corresponds to a distinct direction, whereas each        possible line through a location corresponds to a distinct        linear direction. Just as the available directions are the same        for all locations, the available linear directions are the same        for all locations.    -   coincidence—Two geometric elements of a coordinate system are        “coincident” if, and only if, all of the locations that one of        the elements contains are also contained in the other element.        For example, a line and a vector are coincident if the vector's        starting and ending locations are both on the line. Two        geometric elements that are coincident are said to “coincide.”    -   point—A “point” is a location paired with a time. Two points are        “equal” or “identical” exactly when their locations and times        are both equal. The “distance” between two points is the        distance between the locations of the two points. A point is        “older” than and “precedes” another point if, and only if, its        time precedes the time of the other point, and it is “newer”        than and “follows” another point if, and only if, its time is        after the time of the other point. The specification refers to        the conventional geometric notion of a point as a “geometric        point.”    -   jump—A “jump” is a pair of points, one of which is called the        “starting point,” the other of which is called the “ending        point.” A jump is said to “jump from” its starting point and to        “jump to” its ending point. It is also said to “originate” at        its starting point and to “terminate” at its ending point. The        “vector of a jump” is the vector that starts at the location of        the jump's starting point and ends at the location of the jump's        ending point. The “angle of a jump” is the angle of the jump's        vector. The “length” of a jump is the length of this vector. The        “start time” of a jump is the time of the jump's starting point.        The “end time” of a jump is the time of the jump's ending point.        A jump “precedes” or “follows” another jump if, and only if, the        jump's start time precedes or follows the other jump's start        time, respectively. The “duration” of a jump is its end time        minus its start time.

Although this specification describes orientations, directions, anddifferences between orientations and directions in terms of angles, theinvention is not constrained to representing these notions as angles.Angles are a uniform measure for representing any of these notions andtherefore serve as a convenient unit for generically expressing ways toimplement the invention. There are many representations than cansubstitute for angles, and the invention could be implemented in termsof any of these representations, depending on the constraints of thegesture system. For example, every vector has an associated directionangle, allowing a vector to represent a direction. A slope can alsorepresent a direction whose angle that falls within a known range of πradians. Coordinates can also represent angles, such as by having unitcoordinates on a circle represent the angle of a vector that starts atthe origin and ends at the coordinates. Anywhere this specificationdiscusses angles, equivalent expressions exist in terms of other units.

7.2.3 Path Definitions

These definitions characterize the path of a finger on the gesturesurface.

-   -   touch location—A “touch location” is simply a location on the        gesture surface that a finger is touching or has touched.        Normally the position that a device reports for a finger on a        facet is a geometric point, but a device may instead provide an        indirect specification of the position. For example, a device        may instead report the area of the surface touched. In these        cases, the touch location is a location on the gesture surface        selected from the area to which the finger touch maps. For        example, the touch location can be the point at the center of        the area touched.    -   touch point—The “touch point” of a finger is a pairing of the        finger's touch location with the time at which the finger was at        the touch location.    -   initial touch point—The “initial touch point” of a finger is the        touch point at which the finger first touches the gesture        surface after being in a state of not touching the gesture        surface. The “initial touch location” of a finger is the        location of the finger's initial touch point. The “initial touch        time” of a finger is the time of the finger's initial touch        point. A finger is said to “initially touch” the gesture surface        at its initial touch point.    -   path—A “path” is either a list of locations available to a        finger that travels while touching the gesture surface, or it is        a list of all touch points so far reported for a finger since,        and including, the finger's initial touch point. The latter list        is ordered chronologically starting with the initial touch        point. A path of at least two points is equivalent to a        partially ordered set of jumps. This set is the set of all jumps        that originate at a point on the path and that terminate at a        following point on the path. Consequently, given any two jumps        in the set having distinct starting points, one jump precedes or        follows the other. Jumps having identical starting points are        not ordered relative to each other. A “jump of a path” is a        member of this partially ordered set of jumps for the path.        Absent context specifying otherwise, a singular reference to        “the path” of a finger refers to the most recent path detected        for the finger. Because the times of the points on a path        effectively order the points, a path can be treated as either a        series or a set. A gesture system need not represent a path as a        set of points, however. For example, a gesture system could        instead separately receive and process each point of a path,        discarding previously received points after each new point or        after each group of new points arrives.    -   most-recent point—The “most-recent point” of a finger is the        last point in the finger's path. Similarly, the “most-recent        location” and “most-recent time” of a finger are the location        and time of the last point in the finger's path, respectively.    -   final touch point—The “final touch point” of a finger is the        most-recent point of the finger at the time the finger lifts        from the gesture surface. The “final touch location” of a finger        is the location of the finger's final touch point.    -   finger identity—The “identity” of a finger is a name that the        gesture system assigns to a finger that touches the gesture        surface. It is most pragmatic to use the name of the finger on        the user's hand that the gesture system presumes the user to be        using. For example, the finger identity might take the name        “thumb,” “index finger,” “middle finger,” “ring finger,” or        “pinky finger.” However, depending on the touch device, the user        may be free to use a finger different from the one that the        gesture system assumes the user to be using, in which case the        name that the gesture system assigns to the finger will not be        the name of the finger that is actually performing the action.        For example, a user may gesture with the index finger in a way        that conforms to gesture system's expectations for the middle        finger. In this case, the gesture system will identify the        finger as the middle finger and not the index finger. When the        gesture system supports the use of multiple hands, the finger        identity may also include an indication for whether the gesture        system presumes the finger to belong to the user's left or right        hand. This specification generally equates “fingers” with        “finger identities,” except in discussion involving the        assignment of finger identities.    -   touch history—The “touch history” of a finger is the set of        paths that the gesture system associates with a finger for a        particular gesture. If the gesture system assigns a finger a        finger identity within the context of a gesture, the finger's        touch history is the set of paths that the gesture system has        received and associated with that identity during the gesture.        The gesture system may also associate multiple paths with a        single finger without knowing the finger's identity. For        example, the gesture system may interpret multiple paths as        belonging to the same finger if they are input in the same        vicinity or region of the gesture surface, such as when        receiving a “touch count” (as explained later). It is also        possible for the gesture system to be able to identify the        finger identities to which the gesturing fingers correspond        without being able to determine which finger has which identity.        Even so, each finger still has a touch history possibly        including multiple paths. The last path in a touch history is        called the “most recent path” of the touch history. However, for        the same reason a gesture system need not represent a path as a        set of points, the gesture system need not represent a touch        history as a set of paths. As with points on a path, the gesture        system could instead receive and process each path separately,        discarding previous paths after each new path or each group of        new paths arrives. The gesture system only needs to collect the        information that is relevant to the finger-mapped gesture.

7.2.4 Directional Partitionings & Discrete Directions

A “directional partitioning” is a partitioning of a set of directionsinto at least two ranges of directions. It is an abstract concept thatprovides a generic way to identify the directions in which fingers moveat the granularity of direction required for determining user requests.Each range of a directional partitioning is called a “discretedirection” and contains at least one direction. Every directionalpartitioning includes at least the two discrete directions that thisspecification identifies by the capitalized names FORWARD and BACKWARD.FORWARD represents directions including finger extension away from thepalm, and BACKWARD represents directions including finger flexion towardthe palm. Directional partitionings serve as generalized maps forassigning discrete directions to the directions that the gesture systemderives from gestures.

Direction angles are a generic way to represent directions on thegesture surface, so as a matter of convenience, this specificationrepresents directions as direction angles. A directional partitioningmay therefore be thought of as a partitioning of the direction angles(−π, π] into two or more discrete directions. The directions of adiscrete direction need not be continuous, allowing a single range ofdirections to span the direction angle π without π necessarilydelimiting one end of the range. Although this specification representsdirections as angles, the invention is not constrained to implementingdirections as angles. A gesture system can represent directions asslopes, as unit coordinates on a circle, and as vectors, but far simplerrepresentations are possible when only a few directions are availablefor input. Similarly, there is no requirement that discrete directionsbe represented as ranges of angles. A discrete direction abstractlyrepresents a range of directions according to the representation ofdirection employed, but once the gesture system identifies a discretedirection, the gesture system can subsequently represent the discretedirection in any form, including as a range, as a software object, or asa simple identifier.

FIGS. 25A-25D show some directional partitionings that are useful forassigning discrete directions to fingers as they gesture. Eachdirectional partitioning has an origin at one of the locations 410-413and a “center-forward” direction represented by rays 414-417. Eachorigin 410-413 represents a location on the path of a finger for which adiscrete direction is being determined. Each center-forward direction414-417 is shown as a solid-line ray having an angle that bisects theFORWARD range 418, 420, 424, or 428. It serves as a standard directionfor the directional partitioning, but any standard direction relative tothe directional partitioning can be chosen. A “center-forward angle” isan angle representation of the center-forward direction. FIGS. 25A-25Ddepict the following kinds of directional partitionings:

-   -   Half-Plane Partitioning (FIG. 25A)—A “half-plane partitioning”        splits a plane into two halves 418-419. It contains two discrete        directions, each spanning an arc of π radians. The most useful        half-plane partitioning partitions the plane along the        horizontal, at angle 0 (or equivalently, π). Angles in the half        plane 418 above the horizontal are considered FORWARD, and those        in the half plane 419 below are considered BACKWARD.    -   Bow-Tie Partitioning (FIG. 25B)—A “bow-tie partitioning”        partitions a plane into four pie-shaped ranges 420-423, with        opposite ranges 420-421 or 422-423 spanning the same length of        arc Δ_(FB) or Δ_(LR), respectively. By varying the arc length,        the ranges can be narrowed or broadened to emphasize the        detection of the FORWARD and BACKWARD ranges at the expense of        the LEFT and RIGHT ranges, or vice versa.    -   Quarter-Plane Partitioning (FIG. 25C)—The “quarter-plane        partitioning” is perhaps the most useful bow-tie partitioning        for identifying a gesture's direction. Each of its ranges        424-427 is an arc of π/2 radians. It is useful for allowing        gestures performed in any of four directions to be indicated        with equal ease.    -   Half-Symmetric Four-Way Partitioning (FIG. 25D)—A        “half-symmetric four-way partitioning” is a directional        partitioning that has four discrete directions 428-431 and that        is only bilaterally symmetric, with the FORWARD and BACKWARD        ranges 428-429 spanning different arc lengths. This partitioning        is useful for making either FORWARD or BACKWARD less detectable        than the other.

A discrete direction may be determined from a directional partitioningby comparing the directional partitioning with a direction and readingthe discrete direction that contains the direction. This specificationcalls this direction the “sampling direction,” which when expressed asan angle is the “sampling angle.” However, a given directionalpartitioning may be used as a template for determining directions atvarying center-forward directions. This can be done by effectivelyrotating the directional partitioning to a particular center-forwarddirection before reading the discrete direction at the samplingdirection. The general process for determining a discrete direction istherefore to choose a directional partitioning, to choose a targetcenter-forward direction, to rotate the directional partitioning so thatits center-forward direction is the target direction, and then tocompare the rotated directional partitioning with a sampling directionto read the discrete direction for the sampling direction. The targetcenter-forward direction is called the “reference direction” or else the“reference angle” when the direction is expressed as an angle. An“oriented partitioning” is a directional partitioning that has beenoriented to a reference direction.

The examples in FIGS. 25A-25D can be thought of as template directionalpartitionings each having a center-forward angle of π/2. FIGS. 26A and26B illustrate the generic procedure for determining a discretedirection from a directional partitioning given a reference angle θ anda sampling angle ϕ. FIG. 26A shows the procedure for a half-planepartitioning 447, and FIG. 26B shows the procedure for a quarter-planepartitioning 448. In each figure, the directional partitioning 447-448is rotated around the origin 435-436 to align the center-forwarddirection 437-438 with the reference angle θ. This process makes therotated center-forward direction equal the reference direction and iscalled “orienting” it to the reference direction. The sampling directionis shown as a ray 439-440 that extends from the origin 435-436 at thesampling angle on the oriented partitioning. Visually, the directionindicated is the discrete direction 441-446 of the oriented partitioning447-448 that contains the sampling direction 439-440; FIG. 26A indicatesthe FORWARD range, and FIG. 26B indicates the LEFT range.

The visual representations of FIGS. 26A and 26B depict a simplemathematical relationship among the relevant angles. The discretedirection that a directional partitioning indicates for a sampling angleis the range of angles that contains the angle calculated as follows:add the sampling angle ϕ to the predetermined center-forward angle andsubtract the reference angle θ. Notice that this computation does notactually require rotating the directional partitioning, as the samplingangle and the directional partitioning need only be oriented relative toeach other.

Every pair of FORWARD and BACKWARD ranges of directions has two “sideranges” of directions. Consider the directions that are available toeach side of the FORWARD and BACKWARD ranges. If the FORWARD andBACKWARD ranges are continuous on the left, the left side range isdefined to be the empty set. Otherwise, the left side range is definedto be the range of directions between the left boundary directions ofthe FORWARD and BACKWARD ranges. Likewise, if the FORWARD and BACKWARDranges are continuous on the right, the right side range is defined tobe the empty set; otherwise, the right side range is defined to be therange of directions between the right boundary directions of the FORWARDand BACKWARD ranges.

Suppose S and T are opposite ranges of directions, with one of thembeing the FORWARD range and the other the BACKWARD range, but alsosuppose it is not known which of S and T is FORWARD and which isBACKWARD. The “side ranges” of S and T are defined to be the LEFT andRIGHT ranges of S and T absent indication of which range is LEFT andwhich is RIGHT, and S and T are said to “define” their side ranges. Thisterminology helps to simplify the communication of constraints on leftand right ranges of directions given forward and backward rangesdirections.

This specification uses this generic process for determining directionsbecause the process provides a common framework that works for allexamples. The invention is not limited to this mechanism for determiningdirections, however. Any mechanism for identifying the range ofdirections that contains a particular direction suffices for thisinvention. For example, if a gesture system only recognizes twodirections, such as one towards the top of a touch device and onetowards the bottom, the gesture system need not represent directions asangles, need not compute directions to compare with ranges ofdirections, and need not orient ranges of directions to thecenter-forward direction. Even so, this generic process could be used toimplement the gesture system and hence suffices to illustrate a way toimplement details of the invention. Moreover, by definition, adirectional partitioning represents any partitioning of directions intoranges of directions, so directional partitionings and their discretedirections characterize any gesture system that detects directions.However, there are other approaches to determining a direction'sdiscrete direction, such as by looking up the discrete direction in atable indexed by direction, by dividing a numeric representation of thedirection by the range span of a discrete direction, and byconditionally testing direction range boundaries by program logic.

7.2.5 Local Direction

A finger may change directions intermittently as it performs a gesture,even if only accidentally over short distances. The direction in which afinger is travelling at any point on a path is the “local direction” andcan be given by a direction angle and a discrete direction. Thedirection angle for a point is called the “local angle” for that point,and the discrete direction for a point is called the “local discretedirection” for that point.

There are many ways to compute the local angle for a point. Preciseformulations are possible when many points are available on the finger'spath before and after the point being sampled. However, a gesture systemthat provides real-time feedback to the user about the status of a“direction-constrained” gesture, as explained below, may have toestimate the local angle for the most-recent point, in which case thegesture system can only base the local angle on preceding points of thepath. There are many ways to perform this estimate as well.

A simple way to estimate the local angle for the most-recent point is touse the angle of the jump that ends at the most-recent point and startsat the newest preceding point on the path that has at least a specifiedminimum distance from the most-recent point. This specified minimumdistance is called the “direction assignment distance.” The gesturesystem can implement this approach by keeping a list of all pointsreported for the finger, but this might be taxing for some mobiledevices. As an alternative, the gesture system can keep a list of thenewest points reported for a finger using the following rules:

-   -   1) Every new point reported for a finger is added to the end of        the list.    -   2) Every time a new point is added to the end of the list, all        points are discarded from the beginning of the list that have a        distance from this new point greater than or equal to the        direction assignment distance, except for the newest point in        the list having a distance greater than or equal to the        direction assignment distance, which is kept at the start of the        list.

After performing the above procedure for a new point, the local angleestimate for the most-recent point is the angle of the vector thatstarts at the first point in the list and ends at the most-recent point,which is the last point in the list. This algorithm can further reduceresource consumption by only recording every nth point or by onlysampling points at pre-specified intervals.

There is a reasonable range for the direction assignment distance thatis dependent on the configuration of the host, the sensitivity of thedevice facets, and the resolution of the device facets. When thedirection assignment distance is too small, random accidental shiftingor rocking of the finger registers as changes in direction. When thedistance is too big, the user has to move the finger excessively beforethe gesture system registers the change in direction. There may also bea component of user preference, suggesting that it be configurable.

Finally, the local discrete direction at a point on a path can becomputed from the point's local angle using a previously establisheddirectional partitioning and reference angle. With the directionalpartitioning oriented to the reference angle, the local discretedirection is the discrete direction indicated for the partitioning at asampling angle equal to the local angle.

7.2.6 The Gesturing Threshold

Touch devices can be sensitive and report slightly shifting or rockingfingers as changing locations. In a finger-mapped gesture system theuser ideally types quickly by inputting many small gestures in rapidsuccession. It is helpful for the gesture system to distinguishintentional from unintentional finger movement. Because finger-mappedgestures tend to be very short, the challenge is mainly in determiningwhether a finger is gesturing at all. Fingers may accidentally touch thegesture surface, a user may touch a finger and lift it before performingthe gesture in order to rescind the gesture, or on gesture systems thatsupport “holds,” a user may touch the surface with the intention ofperforming a function analogous to pressing the shift key on a keyboardin order to qualify a gesture concurrently performed by other fingers.To distinguish between gesturing and non-gesturing fingers, the gesturesystem should employ a gesturing threshold.

A “gesturing threshold” is a condition under which the gesture systemconsiders a finger to be gesturing at a point on the finger's path. Afinger “exceeds” the gesturing threshold at the point where it satisfiesthis condition. Once a finger exceeds the gesturing threshold, thefinger is considered to be “gesturing,” and it is considered to begesturing for the remainder of the finger's path, until the finger liftsfrom the gesture surface, unless the gesture system subsequentlyre-designates the finger otherwise. Note that a gesturing threshold isnot the only way that a gesture system can distinguish gesturing andnon-gesturing fingers; it is also possible for a finger to be“gesturing” in a gesture system that does not implement gesturingthresholds. For example, a gesture system may consider any touchingfinger to be gesturing.

There are at least two approaches to implementing gesturing thresholds:detecting a minimum distance and detecting a minimum speed. A gesturingthreshold ideally includes at least the requirement that the fingertravel a minimum distance or greater. This is the “gesturing thresholddistance.” If the gesturing threshold is merely to be sensitive to anymovement at all, the gesture system can set the gesturing thresholddistance to the minimum detectable distance of movement. However, unlessthe fingers are moving against resistance, it is likely easy for a userto accidentally move a finger short distances at any speed, so thegesturing threshold distance should probably be somewhat greater thanthe minimum detectable distance. The gesturing threshold distancetherefore also provides a minimum gesture length for each finger. Eachfinger may have a different minimum length at which it is capable ofgesturing, so the gesture system could have gesturing thresholddistances vary from finger to finger.

Another option is for the gesture system to recognize gesturing fingersas those that move at a minimum speed or greater. The gesture system canaccomplish this by keeping a list of all points that fall within apolling window. The polling window can be an amount of time, in whichcase the newest point is always added to the end of the list, and uponadding a new point to the end, all points older than the newest point bymore than the polling window time are removed from the start of thelist. If the distance between the oldest and newest points, along withthe difference in time between them, indicate that the finger averagedthe minimum speed or greater, the minimum speed condition is satisfied.The speed can also be measured by using a polling window that is adistance, in which case the gesturing threshold requires both a minimumdistance and a minimum speed. Under this approach, the list only includepoints within a predetermined distance of the newest point, plus thefirst point at that distance or greater. The time-based polling windowmonitors for a minimum average speed over a given period of time, andthe distance-based polling window monitors for a minimum average speedover a given distance. It is less desirable to measure only over awindow of time, because to distinguish intentional gestures from fastaccidental movements, the gesture would have to last at least thislength of time, potentially reducing the maximum rate of gesture input.

The straightness of the finger's path is another possible condition forthe gesturing threshold. The gesture system could require that the userbegin a gesture by moving the finger in approximately a straight line. Asimple way to test the straightness of a path between two end pointsfirst determines the distance between the two end points. Call thisdistance D. The test then examines each intervening point in turn andsums the two distances from this intervening point to each end point. Ifthis sum is greater than D by a predetermined percentage of a D, thepath is not considered to be straight. 5% to 10% are reasonablepercentages. This test is equivalent to requiring that all pointsbetween the two end points fall within an ellipse having those endpoints as foci. Testing for straightness can help to reduce theeffective gesturing threshold distance.

Regardless of the approach taken, there is a point at which a finger isdetermined to have exceeded a gesturing threshold and a point at whichthe finger is determined to have begun gesturing. The latter point iscalled the finger's “starting point” for gesturing. Depending on thegesturing threshold in effect, the starting point may not be known untilthe gesturing threshold is exceeded. For example, the starting point maybe the point that precedes the point of exceeding gesturing threshold bythe distance in time or space given by a polling window. A vector fromthe starting point to the point of exceeding a gesturing threshold isthe “gesturing threshold vector.”

The most effective gesturing threshold appears to use a gesturingthreshold distance and at least one additional condition. It issufficient to monitor for a minimal speed over a gesturing thresholddistance. A jittery finger may move quickly over short distances, so itis best to establish a non-trivial gesturing threshold distance. Thisapproach ignores the distance that a finger travels before the userappears to begin gesturing, allowing the user to hesitate beforegesturing without consequence while the finger is touching the gesturesurface. Testing for an “initial slide” is an effective example of thiscombined test.

7.2.7 The Initial Slide and Sliding

Testing for an “initial slide” appears to provide an ideal gesturingthreshold. The “initial slide” of a finger is the first jump on thefinger's path that has at least a specified minimum length and at most aspecified maximum duration. The specified minimum length is called the“minimum initial slide length” and is the gesturing threshold distance.The specified maximum duration is called the “maximum initial slidetime.” The initial slide is therefore the first occurrence of a fingermoving at least a certain distance at a minimum speed.

The characterization of an initial slide precludes more than one jump ona finger's path from satisfying the criteria; each finger has at mostone initial slide between the time it touches the surface and the timeit lifts from the surface. A finger has no initial slide if no jumpsatisfies the criteria.

A finger begins “sliding” the instant an initial slide exists for afinger. An initial slide occurs the instant an initial slide isdetectable, even though this instant happens at the end of the period towhich the initial slide pertains. The “starting point” of an initialslide is the starting point of the initial slide jump, and the “startinglocation” of a slide is the location of the starting point. The “initialslide vector” is the vector that starts at the location of the jump'sstarting point and ends at the location of the slide's ending point;this is the gesturing threshold vector of a gesturing threshold thattests for an initial slide.

A finger is said to “slide” and be “sliding” the instant it has begunsliding, according to the definition of “initial slide.” Moreover, oncea finger has begun sliding, the finger is thereafter considered to slideand be sliding until the finger lifts from the gesture surface. To beclear, once a finger has begun sliding, it is considered to remainsliding no matter how fast or far the finger moves and is sliding evenif the finger is not moving at all.

FIG. 12 depicts the hypothetical path of a finger from the time itinitially touches the gesture surface at the initial touch point 150.The dots 150-159 indicate successively provided points on the path, andthe graph 163 depicts their relative locations. The “T=” numberassociated with each point shows a POSIX integer time for the finger atthat point. In this example, the finger touches point 150 at POSIX time107. The gesture system monitors all possible jumps within either awindow for the minimum initial slide length or a window for the maximuminitial slide time. Here the minimum initial slide length is 6 units ofthe graph 163, and the maximum initial slide time is 5 POSIX units oftime. The first jump to equal or exceed the minimum length in no morethan the maximum time is the jump from point 154 at POSIX time 118 tothe point 157 at POSIX time 123. This jump has a length just over 6units and a duration of 5 units. The gesture system therefore identifiesthe initial slide 160 at time 123; the finger exceeds the gesturingthreshold at point 157. This establishes point 154 as the starting pointof the slide 160 and point 157 as the ending point of the slide 160.

7.2.8 Distance Measure

A “distance measure” is a function that produces a distance between twopoints for some way of measuring the distance. One point is called the“sampling point” and the other point is called the “measurement origin.”The measurement origin remains fixed as the sampling point changes, andthe distance between the two points is measured as a function of thepath of the finger between the points. The sampling point is normallythe most-recent location of a finger.

The three most useful measurement origins are a finger's initial touchpoint, the starting point of a gesturing threshold, and a point at whicha finger reverses direction. In the first case, a distance measure isdefined for the finger from the moment the finger touches the gesturesurface until the moment it lifts. In the latter two cases, the distancemeasure is not defined until certain conditions are met. Once a distancemeasure is defined for a touching finger, the distance measure remainsdefined for the finger until it is redefined for a new distance measureor until the finger lifts.

The simplest distance measure is the length of the distance vector. The“distance vector” is the vector that starts at the measurement originand ends at the sampling point. The distance vector is also called thedistance vector “for” the sampling point, and the angle of a distancevector is also called the “distance vector angle.” The length of thedistance vector is the linear distance between the measurement originand the sampling point. This distance measure is sufficient forfinger-mapped gestures that only require the user to move the fingershort distances. The ideal distance measure uses the actual distancethat the finger travelled, provided that the finger travelled in roughlythe same direction for the entire distance. This is ideal because usersdirectly sense and control the distance the finger actually moves. Underthis second approach, the gesture system sets the distance measure tothe length of the path from the measurement origin to the samplingpoint.

FIGS. 12 and 13 depict distance measures based on the starting point ofan initial slide. In FIG. 12, the measurement origin is point 154. Thedistance measure is undetermined on the portion 161 of the path prior topoint 154, and it is determined on the subsequent portion 162 of thepath, starting with point 154. FIG. 13 shows the path of a finger thattouches at point 165, travels along a line through points 166-174, andfinally lifts at point 175. The finger meanders slowly from point 165 topoint 167 and then moves quickly from point 167 through points 168-170.Upon reaching point 170, the finger has begun sliding, here representedwith the initial slide vector 176. Point 167 is the starting point ofthe initial slide and serves as the measurement origin in this example.The vectors 176, 177, and 178 are distance vectors; the distancemeasures at points 170, 172, and 175, are the lengths of the vectors176, 177, and 178, respectively. The distance measure of the finger atthe time it lifts is the length of vector 178. The other features ofFIG. 13 are described later in the specification when relevant.

Another approach to measuring distance is to take the vector that startsat the measurement origin and ends at the sampling point and project itonto another line and use the length of the projection as the distancemeasure. This line is called a “reference line.” The reference line forthe projection is chosen to have a direction equal to the direction atwhich the finger naturally moves forward. The projection thereforemeasures the distance that the finger travels along the center-forwardor center-backward direction, diminishing the measured distance thefarther the finger strays from one of these directions. It is an optionfor the gesture system to choose the reference line onto which toproject the line segment as a function of the finger's discretedirection. This allows the gesture system to measure the projecteddistance travelled along the center-left and center-right directions aswell: distance measures for left and right movement could be determinedby projection onto a line perpendicular to the reference line used tomeasure center-forward and center-backward distances. More specifically,the distance measure can be determined by projection onto a line thatbisects the range of the discrete direction in which the finger isdetermined to be travelling.

7.2.9 Extent Vectors

It is possible to model the path of a finger as a series of vectors thateach represent a finger's movement in roughly a single direction, if thepath has at least two points. These vectors are called “extent vectors.”Each reversal in direction of the finger corresponds to an extentvector. If a gesturing finger never reverses direction before lifting, asingle extent vector models its path. If a gesturing finger travels backand forth before lifting, the finger's path can be modeled as a zigzagof line segments in which each leg of the zigzag is an extent vector. Anextent vector represents the full extent that a finger travels in adirection that is within π/2 radians of the direction in which theextent vector began. While efficient finger-mapped gestures involve thefingers only moving in two opposite directions, it is unlikely that afinger will exactly retrace a preceding path upon reversing direction,so the gesture system needs to employ an algorithm to detect extentvectors.

To recognize each successive extent vector, the gesture system monitorsthe finger for reversals of direction relative to the directiondetermined for the preceding extent vector. This can be done via angleestimates for each extent vector, determined at the start of the extentvector. The angle estimate for the first extent vector is the angle ofthe vector of the finger's gesturing threshold vector. To monitor forreversals, the gesture system can orient a half-plane partitioning tothe angle estimate, using the angle estimate as the reference angle. Afinger can be considered to enter the reversal state the first time itmoves from one point to another in the BACKWARD range of thisdirectional partitioning.

Once a finger has entered the reversal state, the gesture system canmonitor the finger's distance from the point where it reverseddirection, a point called the “potential end-of-vector point.” If thisdistance ever exceeds a minimum extent vector length, without the fingerhaving left the reversal state, the gesture system records the precedingextent vector so that it ends at the potential end-of-vector point. Italso begins a new extent vector at the potential end-of-vector point,estimating its angle as the angle of the vector from the potentialend-of-vector point to the point where the finger exceeded the minimumextent vector length. This new angle can be used as the reference anglefor a new oriented partitioning for monitoring reversals of the newextent vector. Upon beginning the new extent vector, the finger is nolonger considered to be in a state of reversal.

The finger may reverse again while in the reversal state but beforeexceeding the minimum extent vector length. In fact, the finger mayreverse any number of times before achieving a distance of the minimumextent vector length from the potential end-of-vector point. If duringthese reversals the finger ever travels to a point where its extentvector would have a length greater than what it would have at thepotential end-of-vector point, the finger is considered to be movingforward again and is no longer in a state of reversal. In thissituation, the reversal was too short to result in a new extent vector,and the current extent vector instead lengthens as a result of thefinger having reversed the reversal. Upon leaving the reversal state,the gesture system ceases to monitor the finger for exceeding theminimum extent vector length. A series of reversals that fails to travelfar enough to trigger another extent vector likely corresponds to randomrocking movements of the finger.

The first extent vector starts at the finger's initial measurementorigin. Each time the finger reverses direction and starts a new extentvector, it is an option for the gesture system to establish a newmeasurement origin at the reversal point that starts the new extentvector. Regardless, the gesture system must monitor the length of eachextent vector separately, in order to detect the resumption of forwardmovement after a reversal. The above procedure can be made moreefficient by only testing for reversal when the extent vector reduces inlength from one point to the next, eliminating the need to compute adirection angle between every two points on the finger's path. The lastextent vector ends when the finger lifts, at the location where thefinger lifts.

FIG. 22 illustrates this procedure for determining extent vectors usinginitial slides. The dots 330-351 represent points on the path of afinger, with dotted lines connecting them to indicate their sequence ofoccurrence. The three long bold vectors 362-364 represent the computedextent vectors. The short vectors 359-361 represent the initial vectorsthat establish the estimated angles for the extent vectors 362-364. Eachlong-dashed line 352-354 represents the half-plane boundary between thetwo discrete directions of a half-plane directional partitioning that isoriented to an extent vector's estimated angle. In other words, each ofthe half-plane boundary lines 352-354 partitions the plane in halfthrough the point at its center, therefore also partitioning the rangesof directions of vectors that originate at this center point into two180-degree ranges. The arcs 355, 356, 357, and 358 indicate distancesfrom the points 336, 343, 346, and 343, respectively, with arcs 356 and358 indicating two different distances from the same point 343. Noticethe following:

-   -   The finger begins gesturing at point 330, and an initial slide        is detected at point 331, with vector 359 being the initial        slide vector. Vector 359 establishes an estimated angle for the        first extent vector 362, which has not yet been determined. A        half-plane directional partitioning 352 is oriented to the angle        of this vector 359. Line segment 352 represents the angle of the        boundary between the two discrete directions of this oriented        partitioning. The finger continues gesturing after point 331.    -   The direction angle of a vector from point 335 to point 336        yields the same discrete direction as the initial slide vector        359, according to oriented partitioning 352. It does not produce        a reversal. However, the direction angle of vector from point        336 to point 337 yields a different discrete direction for this        oriented partitioning 352, which has been redrawn at point 336;        this can be seen visually as the vector from 335 to 336 pointing        towards one side of the dashed line 352 and the vector from 336        to 337 pointing towards the other side of the dashed line 352.        Upon reaching point 337, the finger enters a state of reversal,        and point 336 becomes the potential end-of-vector point.    -   The arc 355 represents the distance from the potential        end-of-vector point 336 that the finger must travel in order to        complete the first extent vector and start the second. It is the        distance of the minimum extent vector length. Points 337 and 338        have not reached this distance. Point 339 exceeds this distance.        At point 339, the gesture system establishes the first extent        vector as a vector 362 from point 330 to point 336. Point 336        becomes the starting point of the second extent vector 363,        which has not yet been determined. The vector 360 from point 336        to point 339 provides the estimated angle of this second extent        vector 363. A half-plane directional partitioning 353 is        oriented to this estimated angle, shown as a dashed line        representing the boundary angle between the two discrete        directions of this oriented partitioning.    -   The finger continues gesturing through point 343. The next point        reported to the gesture system for the finger is point 344. The        vector 361 from point 343 to point 344 indicates the opposite        discrete direction from that indicated by the initial vector 360        for the extent vector. Point 343 is therefore a potential        end-of-vector point. Point 344 is at a distance from point 343        greater than the minimum extent vector length, shown as being        outside the arc 356. Hence, point 344 also establishes a new        vector 361. The gesture system records the second extent vector        363 as starting at point 336 and ending at point 343. Point 343        starts the third extent vector 364, which has not yet been        determined. A half-plane directional partitioning 354 is        oriented to the initial estimated angle of the third extent        vector 364, given as the angle of the vector 361 from point 343        to point 344.    -   Upon reaching point 347, the finger is found to have reversed        direction again. Point 346 is the potential end-of-vector point.        The arc 357 indicates a distance of the minimum extent vector        length from point 346, and point 347 has not reached that        distance. The finger is in a state of reversal at point 347, but        no fourth extent vector has been detected. The finger reverses        direction again at point 348 and again at point 349. However,        none of these points passes arc 357 and therefore no fourth        extent vector is triggered. Moreover, none of these points is at        a greater distance from the start of the current extent vector,        point 343, than the potential end-of-vector point 346 is. Arc        358 represents that distance. The finger is in a state of        reversal at all of these points, and none has travelled far        enough to end the state of reversal.    -   Prior to reaching point 350, the finger is in a state of        reversal, but upon reaching point 350 the finger exits the state        of reversal and resumes the third extent vector without        producing a fourth extent vector. At point 350, the finger has        both travelled a distance greater than the minimum extent vector        length 357 from point 346 and achieved a distance from point 343        greater than the distance 358 that point 346 is from point        343—it has passed both arcs 357 and 358. The former condition        would normally trigger another extent vector, but the latter        condition overrides this condition and resumes the current        extent vector. Passing the distance at arc 358 is sufficient to        resume the third extent vector. The series of reversals between        points 346 and 350 did not result in starting a fourth extent        vector.    -   The finger lifts at point 351. Lifting terminates the current        extent vector 364. The last extent vector 364 therefore starts        at point 343 and ends at point 351. The finger has traced the        three extent vectors 362-364.

7.3 Finger-Mapping Posture

Finger-mapped gesturing is optimized for the rapid sequential input ofgestures by a single hand. Input appears to be most rapid when the userholds the hand stationary and selects user requests simply by movingfingers forward and backward on the device facets. “Forward” movement ofthe finger is finger extension, where the fingertip moves away from thepalm, corresponding to the FORWARD discrete direction. “Backward”movement of the finger is finger flexion, where the fingertip movestowards the palm, corresponding to the BACKWARD discrete direction.Backward movement of a finger curls the finger, and forward movementuncurls the finger. Movement of the finger either forward or backward iscalled “longitudinal” movement.

The ideal posture for a hand performing finger-mapped gestures is calledthe “finger-mapping posture.” To envision the posture, first imagineboth hands on a conventional keyboard with the fingers on the home row.That is, on a QWERTY keyboard, the fingers of the left hand are on“A-S-D-F”, and the fingers of the right hand are on “J-K-L-;”. Thefingers are also slightly curled, with the thumbs held in the air.That's the home row posture on a conventional keyboard. Now, leaving thehands in place, remove the keyboard and put a touchscreen tablet ormobile phone under either hand in any orientation—under just one hand,but it doesn't matter which one. Lift the fingers slightly off thetouchscreen, and the hand is now in finger-mapping posture for a deviceaccepting input from four fingers. Should the device employ fewerfingers, the additional fingers may stray from this position. Thefingers may also relax slightly so that they are no longer perfectly inline as required by buttons on a keyboard. Because touch devices areideally designed to conform to the gesturing tendencies of the hand,this posture—or a similar one—is also likely best posture for deviceshaving multiple facets or curved facets.

FIGS. 3A-3C each illustrate a hand 14 in finger-mapping posture inseveral orientations on a touchscreen device 13. The gesture systememploys up to four fingers in each these examples. As with touch-typingon a conventional keyboard, in order to reduce stress on the wrist, thewrist is best kept straight, at least on touch devices containing only asingle flat facet. The double-headed arrows (e.g. 87-89) in FIGS. 7D-7Nillustrate how fingers might gesture forward and backward while in fromfinger-mapping posture.

Users may find it difficult to push a finger to gesture in the “forward”direction on touch-sensitive surfaces. This pushing action is similar toattempting to point a finger while keeping the finger physically incontact with the surface. While the pushing action will work, it is adifficult technique to master. It is easier for a user to instead thinkof dragging the fingers away from the palm by brushing the very tip ofthe finger against the surface. This way the fingers remain curled whilegesturing, remaining more in finger-mapping posture. This not only easesgesture input but also speeds up the input of successive gestures.

When the user holds a hand in finger-mapping posture for input into atouch device, the user is also holding the hand in a position andorientation relative to facets of the touch device. For example, theuser can place a hand in finger-mapping posture in varying positions andorientations over a touchscreen. This specification refers to acombination of a position and orientation of a hand as a “handposition.” The touch device likely does not represent hand positionsbecause the information is not likely available to the touch device.Even so, different hand positions can produce different gesture input,so touch devices may still need to be designed to accommodate differenthand positions. For example, an input area on a single facet may supporta hand inputting in any of various positions and orientations. It mayalso be that a touch device provides only one hand position, ensuringthat a hand always has the same position and orientation relative todevice facets. A device that constrains fingers to grooves may fall intothis category, as may a device that receives input via a virtual realityglove, despite the ability of the user to reposition the glove anywherein space.

Fingers move longitudinally while gesturing in finger-mapping posture. Apath on the gesture surface that is available to a longitudinally movingfinger of a hand in a particular hand position is called a “longitudinalpath” of the finger in that hand position. FIGS. 37A-37B each depict aninput area and example longitudinal paths 615-618 along which thenon-thumb fingers of a hand might gesture while the hand is infinger-mapping posture. Locations 655-658 represent “rest locations” forthe fingers, where the fingers might touch were they to rest on theinput area while in the posture. FIG. 37A shows that it may be possiblefor the paths of gesturing fingers to extend outside the input area, andFIG. 37B shows that an input area 614 may fully enclose longitudinalpaths. Gesturing mainly entails varyingly moving fingers longitudinally,but the gesture system may also recognize fingers moving in otherdirections, such as left and right, associating different gestureoutcomes with the different directions. Gestures that are notlongitudinal are “lateral” gestures.

7.4 Channel-Based Hand Calibration

A gesture system may detect finger-mapped gestures using acharacterization of the hand in finger-mapping posture. Thischaracterization is called a “hand calibration.” The hand calibrationmay indicate a particular location and orientation for the hand on thegesture surface, but it may also characterize a hand independently ofits orientation and location. Of the variety of possible handcalibrations, the “channel-based hand calibration” is one of thesimplest capable of independently characterizing each finger. This handcalibration specifies a particular location and orientation for a handon the gesture surface, but once established, it can be geometricallytranslated to other locations, rotated to new orientations, and evenscaled to new sizes. Simpler hand calibrations tend to represent subsetsof the channel-based hand calibration, and more complicated handcalibrations can generally be represented as channel-based handcalibrations.

7.4.1 Channel Lines

A channel-based hand calibration approximates longitudinal paths aslines on the gesture surface. As a finger moves forward and backward infinger-mapping posture, the fingertip traces a short path on the gesturesurface. The path may be somewhat curved, but users most quickly inputgestures by keeping them short, allowing each path to be approximated bya straight line. These are the “channel lines.” To best accommodate eachfinger's inclinations, channel lines need not be parallel. Those thatare not parallel will converge somewhere on the gesture surface,indicating that there is an ideal position for the fingers—and hence thehand—along the channel lines, at least when channel lines occur on thesame device facet. The rough proximity of this ideal position is calledthe hand's “home position.” A channel-based hand calibration specifiesthe orientations and positions of the channel lines and the homeposition on the gesture surface. FIGS. 8A and 8B represent channel lines108-111 for the index, middle, ring, and pinky fingers. The portions ofthe channel lines shown depict the reach of fingers that start gesturingat locations 104-107 of an input area. FIG. 38 also depicts channellines 663-666 for four fingers of a hand.

While a gesture system may support the simultaneous gesturing ofmultiple hands, this specification refers to a hand calibration as acharacterization of a single hand. The channel lines of adjacentnon-thumb fingers of a hand tend to be roughly parallel. Most usersshould find their hands accommodated by a set of channel lines for whichline angles of every two adjacent channel lines differ by at most π/6radians by class equivalence. Unless the channel lines are actuallyparallel or the fingers all have independent facets, there is a locationon each channel line to which each finger most naturally returns betweengestures, where the finger would touch were it resting on the gesturesurface. These are called “rest locations.” There are also maximumdistances above and below the rest locations that the fingers are likelyto travel while gesturing. These distances bound line segments on thechannel lines. These line segments are called “channel line segments.”FIG. 38 depicts an example hand calibration with rest locations 655-658and channel line segments 659-662.

7.4.2 Finger Calibrations

A “finger calibration” is a representation of the properties of aparticular finger of a hand calibration. A channel-based handcalibration is defined in terms of finger calibrations, representingpossible gesturing fingers. Each finger calibration includes some or allof the following information, including at least an associated fingeridentity:

-   -   Finger Identity—Each finger calibration corresponds to a        particular finger identity.    -   Channel Line—The “channel line” is the line that the finger        traces on average as it moves forward and backward on the        gesture surface while the hand remains in home position. The        line may be theoretical and not actually correspond to a known        longitudinal path of the finger, in which case the line serves        as the presumed average path of the finger. However, the line        may also be computed from actual movements of the finger,        representing a bisection of the space that the finger tends to        occupy as it moves forward and backward, according to the        locations that the finger assumes during its travels, or at        least according to the locations of the maximum extents of the        finger as it travels. A channel line extends infinitely in two        directions, even though the fingers only traverse a portion of        the distance.    -   Channel Line Segment—The “channel line segment” indicates the        maximum extents forward and backward that the finger is likely        to gesture along the channel line when the hand is in home        position. This is not a property that the gesture system must        enforce; rather, it's a property that the gesture system may use        to help accommodate the user. The channel line segment can be        determined for a specific finger of a specific user's hand, but        it can also be assigned by default independently of a particular        user's hand. If it is not explicitly specified, it can be        assumed to be the line segment given by the intersection of the        channel line with the input area.    -   Channel Direction—The “channel direction” is the linear        direction of the channel line. Just as the channel line        represents the average longitudinal path of the finger, the        channel direction represents the average linear direction of a        finger as it travels along longitudinal paths. A channel        direction indicates both a forward direction and a backward        direction along the channel line without necessarily        distinguishing the two directions. Channel direction is        independent of location, so it is possible to establish a        finger's channel direction without also establishing the        finger's channel line.    -   Forward Direction—The “forward direction” of the finger is the        forward direction of the finger along the channel line. It is        one of the two directions of the channel line. The forward        direction of a region may also be thought of as the direction in        which the finger that gestures on the region is most likely to        move when extending. This specification expresses the forward        directions of fingers in terms of angles for illustrative        purposes only and refers to these angles as “forward angles”;        other representations of direction are also possible.    -   Rest Location—The “rest location” of a finger is a location on        the channel line that serves both as an estimate for the        midpoint of the channel line segment and as an estimate for        where the user could rest the finger on the gesture surface to        indicate the placement of the finger for the hand's home        position. Stated another way, the rest location is a location        for the finger when a hand in finger-mapping posture rests all        of the fingers of a hand on the gesture surface. Touchscreen        devices can display finger rest locations to help the user        position the hand in home position for optimal gesturing. The        actual rest locations suitable for a hand in finger-mapping        posture will likely vary over time as the hand gestures, because        the hand is likely to gradually drift across the gesture        surface. Rest locations for adjacent finger identities can        normally be expected to be within 20 mm of each other.

FIG. 38 depicts a channel-based hand calibration for a gesture systemsupporting four non-thumb fingers on a single-facet touch-sensitivesurface. It illustrates rest locations 655-658, channel lines 663-666,channel line segments 659-662, and forward angles (θ_(A), θ_(B), θ_(C),and θ_(D)) for the fingers of a hand in home position. The channel lines663-666 also represent channel directions, with an arrow pointing ineach direction of the channel direction. Notice that the channel linesegments 659-662 appear on the gesture surface in series, one afteranother, so that there is one line segment at each end of the series(659 and 662) and so that each of the remaining line segments (660 and661) is between two other line segments. While it is technicallypossible to support finger mapping with line segments in arbitraryarrangements on the gesture surface, the series arrangement is the mostpragmatic one for input by a stationary hand and hence the one that thisspecification assumes for fingers that share facets in handcalibrations. Were a channel line included in this figure for the thumb,it would likely appear at an angle significantly different from theangle θ_(A) of the index finger's channel line 663, depending on thecharacteristics of the particular user's hand, because the natural pathsof the thumb and the index finger tend to be nearly perpendicular.

7.4.3 Correspondence Series

A “correspondence series” is a group of entities that correspond todistinct finger identities of a hand. The group behaves as a seriesbecause the finger identities are ordered according to the order on thehand of the fingers that they identify. The order of the fingeridentities implies an order for the entities of the group. The actualrepresentation of the group within the gesture system may be unorderedwith each entity associated with a finger identity, but therepresentation of the group may also be ordered, possibly with theentities being implicitly associated with finger identities according totheir order. In any case, a correspondence series is treated as a serieshaving adjacent members, as series do, and having end-of-series members,as series do. Fingers may occur in a correspondence series, as may touchlocations for fingers, paths for fingers, and finger calibrations.

Consider a hand in finger-mapping posture held stationary over atouchscreen. Imagine the fingers brushing forward and backward along thetouchscreen, as the fingers extend and flex. Each finger approximatelytraces a channel line segment. The channel line segments occur orderedon the touchscreen according to the order of their correspondingfingers, as do the associated channel lines. Upon determining the orderin which the channel line segments occur on the touchscreen, the channelline segments and their finger calibrations can be sequentiallyassociated with sequential finger identities, consequently forming acorrespondence series.

When the members of a correspondence series are physical entities,sequential members of the series are physically ordered according totheir order in the series. Adjacent members of a series of physicalentities are physically adjacent, except possibly for buffers betweenthe fingers. For example, as shown in FIGS. 4A-4D, a physical barriercan separate adjacent fingers, and as shown in the facets 51-54 of FIG.6 and their projections 55-58 onto a gesture surface 59, per-fingergesturing areas can be separated from each other by horizontal andvertical space.

Adjacent members of a correspondence series need not correspond to thefinger identities of adjacent fingers of a hand. When they do not, nomember of the correspondence series corresponds to a finger identitythat is between the finger identities to which the adjacent memberscorrespond. There is no requirement that a gesture system only employadjacent fingers of a hand. Moreover, it is possible to describeproperties of a gesture system in terms of subsets of the fingers thatthe gesture system supports. For example, it is possible to describebehavior that is available to a selection of three finger identitieseven though the gesture system supports the use of four fingeridentities. In particular, a subset of a gesture system may itself havethe properties of a gesture system, suggesting that a subset of acorrespondence series of one gesture system can itself be acorrespondence series in the context of another gesture system. In thisscenario, it is possible for physical entities that are part of acorrespondence series to end up serving as buffers between physicalentities of a subset correspondence series.

The use of a correspondence series allows the gesture system to inferunspecified properties of members of the series from propertiesspecified for other members of the series. In order to take advantage ofthis, the correspondence series must include at least three members, andproperties must be specified for at least two of the members. Forexample, when a hand calibration is for three or more finger identitiesand only two finger calibrations are specified, finger calibrations forthe remaining fingers can be approximated through interpolation orextrapolation. FIGS. 46A-46F and 47A-47F, which are explained in detaillater, depict examples of inferring properties of finger calibrationsfrom properties of other finger calibrations. Because thumbs tend tomove and be positioned in ways that are generally independent of otherfingers, this specification sometimes excludes from correspondenceseries entities that correspond to a thumb.

7.4.4 Derived Properties

A number of properties can be derived from the hand calibration for useby the gesture system. Two of them derive from the rest locations of thefirst and last non-thumb fingers in the series of finger calibrationsand are considered to be properties of the hand calibration. Theseproperties are as follows, as illustrated in FIG. 38:

-   -   Orientation Line—The “orientation line” is a line that        intersects the rest locations of the first and last finger        calibrations. FIG. 38 shows an orientation line 667 intersecting        rest locations 655 and 658. It also shows the orientation of the        orientation line as the angle Iii.    -   Hand Breadth—The “hand breadth” is the distance between the rest        locations of the first and last finger calibrations. FIG. 38        shows the hand breadth with length H as the distance between        rest locations 655 and 658.

These properties are mainly useful for re-positioning a hand calibrationon the gesture surface. Hand calibrations can be saved for later use,possibly at different locations on the gesture surface and possibly indifferent orientations. A gesture system that employs parallel channellines only requires the hand breadth property to determine the spacingthat is suitable for a particular user's hand. However, having restlocations for each finger allows the gesture system to betteraccommodate the user's gesturing inclinations. When the channel linesare parallel, the rest locations should be normalized to all occur on anorientation line perpendicular to the channel lines, so that orientingthe orientation line predictably orients the channel lines.

7.5 Finger Identification

Finger-mapped gesturing allows different fingers performing the samegesture actions to yield different gesture outcomes. The outcome of afinger-mapped gesture is a function of the identities of the fingersthat perform the gesture. There are many approaches to establishingfinger identities. On some devices, the host may provide fingeridentities, such as when the device physically tracks the fingers ordistinguishes fingers by their electrical characteristics. On otherdevices, the gesture system may deduce finger identities. When eachfinger uses a different facet of the device, the gesture system canassociate each facet with a distinct finger identity. When multiplefingers share the same facet, the gesture system can implement a “fingermap” to associate fingers touching different regions of the facet withdifferent finger identities. Finger maps are generic enough to beimplemented when the host does not provide finger identities. Regardlessof the approach, in order to maximize user efficiency, the touch deviceshould provide the user with assistance associating fingers with fingeridentities.

7.5.1 Finger Maps and Input Regions

A “finger map” is a partitioning of the input area into two or moreregions, which for contextual clarity may also be called “inputregions.” Being a partitioning, every location in a finger map belongsto exactly one region. Regions do not cross facet boundaries, butmultiple regions may partition a single facet. Regions may correspond toentire facets on multi-faceted devices, but only the facets capable ofserving multiple fingers can be partitioned. According to the definitionof “gesture surface,” different facets are assumed to map to differentareas of the gesture surface, allowing this specification to describethe gesture system as identifying fingers by the input regions that theytouch, or in certain cases involving conflicts, by the proximities offingers to input regions.

When the host does not provide finger identities, the gesture systemidentifies fingers with finger maps. To determine the identity of afinger using a finger map, the gesture system first associates eachfinger that touches the input area with a region of the finger map. Bydefault, a finger is associated with its originating region. A finger's“originating region” is the region that contains the finger's initialtouch location. When multiple simultaneously touching fingers are inconflict by having the same originating region, the gesture system mayreassign the fingers to distinct regions. Except when reassigned toresolve conflicts, a finger retains its region association for as longas it remains touching the gesture surface, no matter where the fingermoves, even if it moves into a different region. A finger also retainsits region association even if it moves out of the input area, for aslong as it remains touching a facet. Each region is associated with afinger identity, and each finger is assigned the finger identity of itsassociated region. Using a finger map, the gesture system thus presumesto know the identities of fingers that touch the gesture surface.

The form of a finger map and its input regions can be tailored tofacilitate associating input regions with finger identities. FIGS. 7A-7Nillustrate some possible finger maps for a single-facet input area. Thedouble-headed arrows (e.g. 87-89) indicate the anticipated directions offinger extension and flexion and can be thought of as channel lines orchannel directions. The dots 90-97 indicate regions supporting “holds,”which are explained later. The finger maps of FIGS. 7A and 7B areexamples of arbitrary partitionings of the input area. Arbitrarypartitionings limit the rate at which users can input gestures becausethey are not design for efficient input. Finger map of FIG. 7C has auniform layout analogous to a keyboard, but a user attempting to inputfinger-mapped gestures on it must be careful to originate verticalstrokes in the proper region 65-72. The configuration would be bettersuited as two distinct finger maps, one for regions 65-68 and one forregions 69-72, that the user selects between while inputting userrequests. The finger maps of FIGS. 7D, 7E, and 7F only supportfinger-mapped gestures of two or three fingers. The finger maps of FIGS.7G, 7H, and 7I each support input by the index, middle, ring, and pinkyfingers. For example, regions 63, 64, 98, and 99 identify the index,middle, ring, and pinky fingers, respectively. Regions 78-81 areparallel in FIG. 7G, making this finger map especially well suited for asmall input area such as might be expected on a touchscreen smartphone,but poorly suited for people whose fingers do not easily gesture alongthese directions. The finger maps of FIGS. 7J, 7K, and 7L includesupport for the thumb performing a hold, as explained later. FIGS. 7Mand 7N show finger maps that are designed to support two hands at once.These latter finger maps can be implemented as two distinct finger mapsworking together. If they are implemented as a single finger map, thefinger identity includes an indication of whether it refers to a fingeron the left or right hand.

The input areas shown in FIGS. 7A-7N have varying shapes, as do theregions (e.g. 65-81) depicted in the input areas. The form of thesurface allocation in part limits the form of its input area and hencealso the forms of the input regions. It is useful to select the forms ofan input area and its input regions to optimize them for a particularperson's hand, according to measurements that are specific to theparticular hand. The contour of the finger map is limited by theboundaries of the surface allocation that the host provides, but agesture system may further limit its extent to just the area that aparticular hand is likely to reach while gesturing.

7.5.2 Hand-Optimized Finger Maps

Gesturing on finger maps having more than two regions is difficult andunreliable unless the finger map is tailored to the requirements of aperson's hand. As with touch-typing on a conventional keyboard, rapidinput requires that a hand remain stationary as the fingers quicklymove. This places each finger in its own region. Even so, people's handshave varying widths, placing their fingers at varying distances apart. Aconventional keyboard requires that everyone place their fingers on thesame keys of the home row, but hands of different sizes have differentlevels of difficulty reaching other keys. Sliding fingers forward andbackward on a gesture surface is analogous to reaching for keys otherthan those on the home row and presents similar difficulties. Toeliminate the difficulties of accommodating different users, a fingermap can be based on a hand calibration.

Consider the finger map in FIG. 7E. This finger map has three regions75-77. To maximize gesturing efficiency, these regions 75-77 should bespaced according to the needs of a particular person's hand, thusselecting the width of the middle region 76. Spacing is not required forfinger maps of only two finger identities, such as the finger map ofFIG. 7D, where a person need only have two fingers straddle the boundary83 between two regions 73-74, obviating the need to space regions 73-74specifically for the hand. A hand calibration provides the informationnecessary to space regions to suit a particular hand.

A hand calibration for a finger map determines the spacing of the inputregions, so that the regions are comfortably located under the user'sfingers. It need only specify the relative positions of regions, ratherthan the actual positions of the regions on the gesture surface, as longas the gesture system provides a means for positioning the resultingfinger map. For example, the gesture system could center the finger mapon the given surface allocation, or the gesture system could provide away for the user to select the placement of the finger map and thenconstruct a finger map at that placement having the indicated relativespacing properties. A channel-based hand calibration is capable ofspecifying the relative positions by virtue of specifying positions ineach finger calibration, but simpler hand calibrations are possible. Forexample, a hand calibration could just specify a width, such as byhaving the user input a number or by spacing two user interface elementsto an appropriate distance apart. This width can be used as a handbreadth, indicating how far apart the end regions are to be separated,with the sizing of the interior regions being determined byinterpolation. This specification later describes many other possiblehand calibrations.

In order to take advantage of the efficiency available to finger-mappedgesturing, a finger map must have at least two regions designed toaccommodate the gesturing of at least two different fingers of the samehand. When the fingers of the same hand reside on different facets, thetouch device itself can ensure that the facets accommodate the fingersfor gesturing, such as by allowing the user to change the relativepositioning of the facets. When multiple fingers of the same hand resideon a shared facet, a hand calibration specifies how to partition thefacet into regions.

Regions of a shared facet that correspond to fingers of the same handform a correspondence series. Some examples might include: a) when afacet accommodates multiple fingers of the same hand, b) a channel-basedhand calibration specifies a series of adjacent finger identities, c) aseries of adjacent channel lines or channel line segments, or d) aseries of adjacent regions for the facet, all corresponding. Ideally,the gesture system defines adjacent regions to be contiguous in order tomaximize the space available to a finger for gesturing and to minimizethe odds of a gesture attempt going unrecognized. When a handcalibration specifies the spacing of regions, the gesture system shouldwiden regions as necessary to keep adjacent regions contiguous. Acorrespondence series of regions typically has the property that astraight line can be drawn on the gesture surface through all of theregions in the series. FIG. 27B depicts this with the dashed line 458through regions 454-457.

7.5.3 Channel-Based Finger Maps

There is a way to base a finger map on a channel-based hand calibrationthat is ideal for its simplicity and for its benefit to users. Theapproach specifies the boundaries between regions that are adjacent on ashared facet. In a channel-based hand calibration, each fingercalibration defines an input region by defining a channel line thatrepresents the average longitudinal path of the finger in that region.As users gesture, their fingers are likely to touch the gesture surfacearound this channel line, so the ideal finger map associates each fingerwith the nearest channel line. This is accomplished by putting theboundary between adjacent regions exactly halfway between their channellines, bisecting the space between the channel lines. This assigns eachlocation of the input area to the region whose channel line is nearestto that location. The locations on the boundary line itself should alsobe assigned to one of the adjoining regions.

The inventor has derived a set of equations for determining the idealboundary line between adjacent regions, given channel lines of theregions' finger calibrations. The first equation yields an indication ofthe distance between two channel lines relative to a particular locationon one of the channel lines. Specifically, the equation yields thelength of a line segment having one terminus at the particular location,having the other terminus on the other channel line, and passingperpendicularly through a line that bisects the area of the gesturesurface between the two channel lines. Let the first channel line begiven by the location (Q_(X), Q_(Y)) and angle θ_(Q). Let the secondchannel line be given by the location (R_(X), R_(Y)) and angle θ_(R).The “distance” d from location (Q_(X), Q_(Y)) to the second channel lineis then given by the following equation, where absolute values aredenoted in vertical bar notation:

d=|(sin(θ_(R))*(R _(X) −Q _(X))+cos(θ_(R))*(Q _(Y) −R_(Y)))/cos(|θ_(Q)−θ_(R)|/2)|

To specify the boundary line between the two adjacent regions, let(P_(X), P_(Y)) be a location on the boundary line, and let θ_(P) be theangle of the boundary line. The boundary line is calculated as follows:

θ_(P)=(θ_(Q)+θ_(R))/2

P _(X) =Q _(X)+(d/2)*sin(θ_(P))

P _(Y) =Q _(Y)−(d/2)*cos(θ_(P))

The gesture system can apply this set of equations to determine theboundary line between every two adjacent regions on a facet defined by aseries of finger calibrations. FIG. 42 depicts some of the geometricproperties of a finger map computed via these equations. In this figure,region boundaries 682-684 are computed for a rectangular input area froma hand calibration for four fingers. The dots 674-677 represent restlocations and the bold arrows 678-681 represent channel lines. Theangles θ_(A), θ_(B), θ_(C), and θ_(D) are line angles for the channellines 678-681. To compute region boundary 682, let (Q_(X), Q_(Y)) and(R_(X), R_(Y)) be the coordinates for rest locations 674 and 675,respectively. Rest locations 674-675 are chosen because they are pointsthat the hand calibration provides for the channel lines 678-679. Letθ_(Q) be the angle θ_(A) of the channel line 678 through point 674, andlet θ_(R) be the angle θ_(B) of the channel line 679 through point 675.The value of the computation for d yields the length of the dashed linesegment 685 through points 674 and 688, although point 688 has not yetbeen computed. d is the shortest distance from rest location 674 tochannel line 679. Computing θ_(P) yields the angle θ_(AB), which is theangle of the boundary 682 between regions 670 and 671. Computing (P_(X),P_(Y)) yields the coordinates of point 688, which is the midpoint of thedashed line segment 685. Point 688 and the angle θ_(AB) together specifya line, which is the boundary line between regions 670 and 671, halfwaybetween channel lines 678 and 679.

After computing all three interior boundary lines 682-684, each point onan interior boundary line is equidistant from the nearest points on thetwo adjacent channel lines. FIG. 42 depicts this property for the points688-690 on the interior boundary lines 682-684. Each point 688-690 isthe midpoint of a dashed line segment 685-687 that is perpendicular to aboundary line 682-684, as illustrated by the lengths U_(AB), U_(BC), andU_(CD) indicated for each half of the line segments 685-687. The lengthsU_(AB), U_(BC), and U_(CD) together with the angles β₂, β₃, and β₄,depict the relationships that establish the boundary lines 682-684 asbisecting the area between adjacent channel lines.

The equations successfully determine a boundary line between twoconverging channel lines as well as a boundary line between two parallelchannel lines. The boundaries can be used not only for determining theidentities of touching fingers, but also for displaying the boundariesto assist the user with positioning the hand. The gesture system neednot actually determine the boundary lines, as the gesture system couldequivalently determine the nearest channel line to a finger's touchlocation and assign the finger the finger identity to which the channelline corresponds.

Channel-based finger maps are able to shape and orient each input regionto the preferences of a particular finger of a particular hand. It maybe that most people are able to perform finger-mapped gestures in inputregions having parallel, vertical boundaries, once appropriately spaced.However, it may also be that a small but non-negligible portion of thepopulation cannot gesture reliably under these constraints: some peoplemay have fingers that insist on gesturing forward or backward atnon-negligible differences in direction. Absent accommodation by thefinger map, the user could frequently indicate the wrong finger identityor the wrong discrete direction, should the gesture system parameterizeuser requests by gesture direction. For example, unless the shape of aregion is slightly angled to accommodate a finger, the finger may onoccasion start a gesture in the wrong input region and thereby indicatethe wrong finger identity.

A distinctive characteristic of channel-based finger maps is that theyare able to capture finger movements that are approximately but notperfectly parallel. Heeding even limited differences between adjacentchannel directions also enable the gesture system to better accommodategestures that distinguish between left and right movement of the hand.If a user naturally moves a finger forward and backward at a large anglecompared to the angles at which other fingers of the same hand moveforward and backward, the gesture system may not always properlydistinguish left-and-right movement of the finger fromforward-and-backward movement. However, if the user performs the left orright gesture by pivoting the hand left or right at the wrist instead ofmoving the entire forearm, the finger moves in a direction that islocally perpendicular to the finger's natural forward and backwarddirections of movement, requiring the finger map to be sensitive to theranges of forward and backward directions that are specific to thefinger. By determining a gesture's direction relative to the directionsat which each finger naturally moves, no matter how minimally differentthose directions are, the gesture system is able to more accuratelydetect an intended LEFT or RIGHT range of movement or other range notFORWARD or BACKWARD.

7.5.4 Multi-Hand Finger Maps

The gesture system may also deduce finger identities for a gesture fromcalibrations for multiple hands, such as hand calibrations for both theleft and right hands. It is possible for a single gesture system tomonitor an input area with a single finger map consisting of regions forfingers of both hands, where each finger identity also identifies theleft or right hand. FIGS. 7M and 7N illustrate such finger maps.However, there is also a way to architect the gesture system so that itscales across arbitrary numbers of coordinating hands. This latterapproach leverages a gesture system designed for only one hand, witheach gesture system employing a single hand calibration. The gesturesystems independently report their gesture outcomes to the host, butalong with each gesture outcome they also report the time at which allgesturing fingers were first touching the gesture surface and the timeat which the first gesturing finger lifted. The host combines theseparately reported gesture outcomes to determine the behavior that theuser intended to indicate. The two times accompanying each gestureoutcome enable the host to determine whether gestures by separate handson separate gesture sessions are joint or sequential: two gestures arejoint exactly when the intervals that these times delimit overlap. Jointgestures together indicate a single user request, while sequentialgestures indicate sequential user requests.

7.5.5 Assistive Association

Consider a touch device that uses the same fixed-width finger map forall users. On this touch device, the finger map never changes, and theuser has no ability to select among different region widths. Such afinger map may be fixed in the form of the finger map of FIG. 7G. Thistouch device delegates to the user all responsibility for properlyindicating finger identities. The user must train each finger to gesturein the appropriate region. When there are only two adjacent regions, thefingers need only straddle the boundary between the regions, but whenthere are more than two regions, the fingers must spread appropriatelyapart, because each finger must be positioned relative to adjacentfingers. If multiple devices employ this approach using finger maps ofdifferent widths, the user must adapt the hand to the varying widthswhen moving from device to device. These devices provide the user withno assistance for indicating finger identities, thereby restrictingefficient, accurate input to skilled and adaptable users. Hence, touchdevices for three or more fingers best serve users by providingassistance associating finger identities with fingers.

There are several ways for a touch device to assist users with inputinto a finger map. Generating a finger map from a hand calibration isone such way. Another is to provide multiple finger maps of varyingwidths to give the user a selection of finger maps to choose from. Theuser can choose the finger map that is most effective for the user'shand. The touch device can also provide the user with finger maps of avariety of forms, where the input regions may also vary in form. Thetouch device may also provide the user with a fixed finger map thatvaries in width, allowing the user to place the hand in home position onthe finger map over the portion of the finger map according to the widththat suits the user. The finger maps of FIGS. 7E and 7I can be used forthis purpose. For example, a child could place the hand with the fingersnear the bottom of the finger map, where the finger map is narrow, whilean adult could place the hand with the fingers near the top of thefinger map, where the finger map is wide. People with hands ofintermediate size could place the hand with fingers somewhere midwaybetween the top and the bottom.

The touch device could also provide physical boundaries between fingersto keep each finger gesturing on its associated input region. FIGS.4A-4D illustrate various placements of a boundary 19-22 between fingers.A touch device may even include a boundary between every two fingersthat can gesture, or equivalently, the device may provide a groove foreach finger to follow in order to reduce the chance that a finger willaccidentally gesture on the wrong portion of the gesture surface. Inthese scenarios, the gesture system may implement a finger map todetermine the identities of gesturing fingers or the host may directlyindicate the finger identity. For example, if the boundaries or groovesrestrict each finger to a distinct facet of the touch device, the hostcan report the finger's facet to the gesture system. The facet indicatesthe finger identity, so there is no need for the gesture system to use afinger map to distinguish fingers of that finger identity. When eachfinger has its own facet, as in FIG. 6, there is no need for a fingermap at all.

A host that determines finger identities for the gesture system may beable to assist the user by making the process of indicating fingeridentities invisible to the user. Compare FIG. 37A, which represents aninput area that is a finger map, with FIG. 37B, which represents aninput area that does not use a finger map. In FIG. 37A, the user mustplace fingers so that they gesture paths 615-618 in the proper inputregions 610-613, whereas in FIG. 37B, the host is expected todistinguish fingers for the user. Assuming that the host can accuratelydistinguish fingers, the solution of FIG. 37B is preferable becausefingers cannot drift out of their input regions.

There are many ways for a host to distinguish fingers. One way is tomechanically track the fingers. A probe or sensor could be fixed to eachfinger, so that the identities of the fingers correspond to theidentities of the probes or sensors that provide the data. Gloves thattrack the movements of fingers fall into this category. The host mayeven track fingers by more sophisticated means, such as by analyzingvisual, sound, or magnetic field information. In these cases, the hostlikely determines finger identities according to heuristics forassociating three-dimensional information with individual fingers. Itmay also be possible for a device to identify fingers according toelectrical properties detected for the fingers as they gesture on atouch-sensitive surface. In this case, the host distinguishes fingers ofdifferent finger identities by their electrical signatures. Someelectrical signatures may be common to many people, while some mayrequire calibration to a particular hand. A device may even combinemethods for determining finger identities. In these cases, it isunnecessary to have multiple regions or facets.

It is also possible for the host to provide some but not all fingeridentities. It may be that the host is only be able to determine theidentities of specific fingers, such as just identities for the thumband the index finger, perhaps because their electrical properties happento be recognizable. In this case, the gesture system can employ a fingermap to determine the identities of the otherwise unidentified fingers.FIGS. 58A-58C each show a device with a facet 960 dedicated to onefinger along with a second facet having a finger map divided intoregions 961-963 for determining the identities of additional fingers.However, even if all of the fingers use the same facet, the gesturesystem need only employ a finger map for the unidentified fingers.

The host provides a finger identity either by reporting it directly tothe gesture system or by reporting a specification from which thegesture system can determine the finger identity. For example, when afacet directly corresponds to a finger identity, reporting the faceteffectively reports the finger identity. However, a host may insteadcollect a complex signature for a finger and defer to the gesture systemthe determination of a finger identity. This also amounts to the hostreporting the finger identity, as the boundary between the host andgesture system is immaterial in this case. All that matters is thatfinger identities become available for interpreting finger-mappedgestures, and ideally, that the touch device assists the user withindicating those identities.

7.6 Finger-Mapped Gesturing

Finger-mapped gestures are gestures that select user requests accordingto the fingers with which the user gestures and according to the actionsthat the user performs with the fingers while gesturing. There are manypossible classes of finger-mapped gestures. The present inventionconcerns itself mainly with the finger-mapped gestures that can be inputwhile keeping the hand stationary and only moving the fingerslongitudinally, as these are the most efficient gestures. It is usefulto constrain finger-mapped gestures to be “lockstep” in direction andlength, in order to teach users to gesture with maximal efficiently andto help disambiguate finger-mapped gestures from other gestures that thehost might support. Non-finger-mapped gestures might include gesturessuch as two-finger pinching and zooming actions, which the host mightallow the user to perform on top of the input area for applicationpurposes. This section describes detecting finger-mapped gestures andassigning finger identities to fingers according to the input regionsthat the fingers touch.

7.6.1 Finger-Mapped Gestures

A “finger-mapped gesture” is a gesture that the user performs on theinput area to input a user request, selecting the user request accordingto the finger identities of the gesturing fingers and the actions thatthe fingers perform. The combinations of gesturing fingers aresufficient to input a large variety of user requests, but it isdifficult to remember many arbitrary assignments of user requests tofinger combinations. Instead of relying strictly on finger combinations,the gesture system introduces at least one additional way to select userrequests so that it can avoid the difficult finger combinations. Theadditional ways to select user requests are based on what the fingers doduring the gesture. For example, finger-mapped gestures already entailthe user moving fingers over a distance, so it is convenient for thegesture system to further select the user request by the direction inwhich a finger moves and the distance that the finger moves in thatdirection, without requiring the user to perform additional gestureactions. Under this approach, the finger identities, direction, anddistance together parameterize the gesture, thus parameterizing thegesture outcome.

The various means for parameterizing a finger-mapped gesture are called“gesture parameters.” Specifically, a “gesture parameter” is anattribute of a gesture that has a particular value for a particularoccurrence of a gesture. At least two values are possible for eachgesture parameter, and the gesture outcome of the gesture is dependenton the value. Notice that the finger identities of a gesture arethemselves a gesture parameter. A gesture parameter that determines adiscrete direction for the gesture is called a “gesture direction.”There are a number of ways to define a gesture parameter based on thedistance that a finger moves relative to a measurement origin. Thesimplest is perhaps the one this specification calls a “gesture level.”Other possible parameters include (a) the number of times the user tapsa finger prior to performing the gesture (“touch count”), (b) thefingers that the user is holding down on the gesture surface to behaveas “shift” keys (“shift holds”), and (c) the distance that a fingertravels after reversing directions (“backbrush level”). The gesturesystem determines a gesture outcome from the gesture parameters andreports user requests to the host.

Parameterizing gestures via gesture parameters allows users to input abroad variety of gestures. A “gesture parameterization,” or simply a“parameterization,” is a set of values that are each associated with agesture parameter of a gesture. Each gesture parameter, except for theselection of finger identities, has the property that the parameter'svalue is largely independent of where the gesture is input on thegesture surface, other than the extent to which the value may depend onthe finger identities. This property is called “position independence”and is explained in detail later. For brevity, a reference in thisspecification to a gesture parameter indicates the value of the gestureparameter when the context of usage requires a value for the gestureparameter.

Each gesture parameterization available to the gesture systemcorresponds to a particular gesture outcome. There are different waysfor a gesture system to report a gesture outcome to the host. Under themost generic approach, the gesture system reports the gestureparameterization itself to the host. The host attempts to map theparameterization to a behavior. It is also possible to employ a gesturesystem that has specific knowledge of how the parameterizationscorrespond to host behaviors. In this case, the gesture outcome candirectly identify the specific behavior desired of the host, such as byname or identifier. These are both examples of the gesture systemreporting a user request to the host. If the gesture system everdetermines that the user is not entering a valid finger-mapped gestureor that the provided gesture parameterization in not one of theparameterizations that it associates with a user request, the gesturesystem need not report the gesture outcome. Alternatively, the gesturesystem may report unrecognized gestures.

There are many ways to represent the value of a gesture parameter inprogram logic. One way is to use a name, number, or other identifier forthe value. This approach allows a parameterization to be represented asa list of values. The gesture system can use lookup tables to identifythe gesture outcome to which any list of parameter values corresponds,doing so by selecting rows and columns within the tables according tothe parameter values associated with the rows and columns. These tablescan be many-dimensional to accommodate many parameters. When processingspeed is important, it is common practice in software to use decisiontrees instead of lookup tables, in which case the program logic neednever actually compute an identifier to represent the value of anygesture parameter. In this case, the program logic simply branches tocode that is specific to the scenario where the gesture parameter wouldotherwise have a particular value. Because the program counter points toa branch of program logic that is specific to a particularparameterization, the branch of program logic can serve as aparameterization for the gesture, thus also representing a parametervalue for each gesture parameter. It is also possible to hybridize theseapproaches. Representing parameter values as software objects canfacilitate the hybridization. In general, a parameter value is theresult of evaluating a gesture parameter for a particular gesture.

7.6.2 Gesture Bases

The finger-mapped gestures that a gesture system supports can be definedin terms of the gesture bases that characterize the gestures. A “gesturebasis” (plurally “gesture bases”) is an association of a non-empty setof finger identities with a non-empty set of gesture parameters. The setof finger identities is called the “characteristic identity set” of thegesture basis, and the set of gesture parameters is called the“characteristic parameter set” of the gesture basis. The characteristicparameter set does not include a gesture parameter for the set of fingeridentities because the characteristic identity set specifies the valueof that particular gesture parameter for the gesture basis. A gesturebasis represents a set of gestures that all use the same set of fingeridentities and the same set of gesture parameters. Each gesture of agesture basis has a parameterization that characterizes the gesture.Each of these parameterizations includes the finger identities of thecharacteristic identity set and values for the gesture parameters of thecharacteristic parameter set. Together these parameterizations are the“selectable parameterizations” of the gesture basis. A selectableparameterization is said to “indicate” the user request that itrepresents. The “path parameterization” of a selectable parameterizationis the portion of the selectable parameterization that consists of allof the parameter values other than the finger identities.

A gesture system implements each gesture basis of a set of gesturebases. The gesture bases of a set are said to be “independent” if, andonly if, no two gesture bases of the set both have characteristicidentity sets that are identical and have characteristic parameter setsin which one is a subset of the other. Given two gesture bases, if acharacteristic parameter set of one were a subset of the characteristicparameter set of the other, they would not represent distinct sets ofgestures. A gesture system is therefore most usefully characterized asimplementing a set of independent gesture bases. Gesture bases withdifferent characteristic parameter sets may have the same characteristicidentity set, provided that the characteristic parameter sets aredistinguishable from the touch histories of the fingers. Note that agesture basis may characterize a gesture that uses more gestureparameters than those in the character parameter set of the gesturebasis; gesture bases need not fully specify gestures. The “series ofavailable finger identities” of a set of gesture bases is a series offinger identities consisting of the set union of the characteristicidentity sets of all the gesture bases in the set of gesture bases, withfinger identities ordered in the series according to the order of thefingers on a hand that they are assumed to represent. A “selectableparameterization” of a set of gesture bases is a selectableparameterization of a gesture basis of the set.

It is possible to have a set of gesture bases that only characterizesgestures that are composed of other gestures of the set of gesturebases. For example, suppose the series of available finger identities isas follows: index finger, middle finger, and ring finger. One gesturebasis may have a characteristic identity set that contains only the ringfinger, another that contains both the ring finger and the middlefinger, and another that includes all three fingers. The gesture basesof this set can potentially be distinguished by number of fingerswithout requiring individual fingers to be identified. This is a form ofgesture “subsumption” and contrasts with “non-subsumptive” sets ofgesture bases. A “non-subsumptive” set of gesture bases is a set ofgesture bases in which all of the characteristic identity sets of thegesture bases are different and yet have the same number of fingeridentities. Moreover, every non-subsumptive set of gesture basescontains multiple gesture bases. A non-subsumptive set of gesture basesdescribes a collection of gestures that all use the same number offingers. Because a set of gesture bases need not characterize all of thegestures available to a gesture system, multiple non-subsumptive sets ofgesture bases may characterize the same gesture system, using onenon-subsumptive set for each number of fingers available to a gesture.

When an input area “implements” a set of gesture bases, the gesturesystem implements the set of gesture bases as gestures that can beinitiated from locations on the input area, and the gesture system isnot constrained to only implementing the indicated set of gesture bases.The gesture system need not represent any given gesture basis as adistinct component. Instead, a gesture system implements a gesture basisby virtue of implementing the gestures that the gesture basischaracterizes. A gesture basis is just an abstract characterization ofgestures available to a gesture system. Similarly, the gesture systemneed not represent a set of gesture bases as a distinguishablecomponent. The series of available finger identities of a set of gesturebases is also just a characterization of the finger identities that thegesture system supports; it is the series of finger identities that thegesture system uses with the gestures that the gesture basescharacterize. For example, a gesture system might implement a set ofgesture bases as a table that returns a set of gesture parameters for agiven set of finger identities, or a gesture system might hardcode theavailable gesture parameters and look up a user request in tables thatcross reference finger identities with the values for the variousgesture parameters. It is also possible to implement a set of gesturebases without tables, such as by having state machine software objectsor programming language “case” statements assign user requests, withprogramming logic identifying the user request.

7.6.3 Detecting Gestures

Each finger-mapped gesture involves the user touching one or morefingers to the gesture surface, moving those fingers, and then liftingthe fingers. During this process, the gesture system may opt todistinguish the movement of gesturing fingers from fingers that are notgesturing. Upon detecting the fingers that are participating in agesture, the gesture system interprets the paths of those fingers ascontributing to a gesture parameterization. It then reports theparameterization to the host in the form of a user request. This sectionintroduces the details of this overall process.

7.6.3.1 Gesturing/Non-Gesturing Distinction

Users interacting with the gesture surface normally either “tap” afinger on the gesture surface, “hold” a finger on the gesture surface,or move a finger along the gesture surface. A user taps or holds afinger to indicate a location on the gesture surface, and a user moves afinger to indicate a path of multiple points. Only movement isconsidered “gesturing”; tapping and holding are considered to be“non-gesturing” interactions with the gesture surface. Because moderntouch-sensitive surfaces are highly sensitive to finger movement, tapsand holds are likely to input multiple points and therefore also input apath that could be interpreted as gesturing. It is an option for thegesture system to make an effort to distinguish gesturing andnon-gesturing according to properties of the paths, allowing the gesturesystem to either ignore non-gesturing or give it meaning. The path of afinger that the gesture system considers to be gesturing is called a“gestured path.”

There are several approaches to distinguishing gesturing andnon-gesturing. Under one approach, any finger touching the gesturesurface is assumed to be non-gesturing until it exceeds a gesturingthreshold, at which point it is assumed to be gesturing. Under anotherapproach, any finger touching the gesture surface is assumed to begesturing until it satisfies a condition that identifies it asnon-gesturing. If the gesture system only needs to recognize fast taps,the non-gesturing condition could be that the finger touches and liftsbefore a timeout period. This condition is suitable for ignoringextraneous touches and for interpreting rapid touch input as gesturing.Alternatively, the gesture system might only recognize a finger asnon-gesturing if it has been touching for at least a certain amount oftime. This condition is suitable for distinguishing an intention topress a user interface button that underlies the input area from anintention to perform a finger-mapped gesture. It is also suitable forinterpreting fingers as resting on the gesture surface without providinginput. Additional constraints ideally embellish these conditions, suchas limiting the length of a non-gesturing path. The gesture system mayalso treat fingers as neither gesturing nor non-gesturing until thefinger satisfies a distinguishing threshold that appropriatelycharacterizes the finger.

The gesturing threshold is the ideal condition, regardless of whetherthe gesture system also implements a non-gesturing condition. Agesturing threshold provides the following benefits, some of which arefurther explained later in the specification:

-   -   Supports ignoring accidental touches    -   Supports “touch count” parameters, which count taps    -   Supports parameters that monitor changes in direction, such as        “backbrush level.”    -   Supports the rapid and temporary indication of “shift holds”        while gesturing.    -   Supports superposing the input area on touchscreen user        interface buttons.    -   Supports ascribing special meaning to gestures strictly        involving taps.    -   Supports allowing the user to rest fingers on the input area        without effect.

An implementation of the gesture system that uses neither gesturingthresholds nor non-gesturing conditions can interpret all touchingfingers as gesturing. Some gesture parameters will require that a fingertravel a path of at least two points, however. It is an option tomonitor for the rare circumstance in which a tap only inputs a singletouch point and to decline to recognize the tapping finger as gesturing,although the value of doing so may depend on the sensitivity of thetouch device.

Gesture systems grow more complicated with the addition of a gesturingthreshold. This specification hereafter illustrates gesture systems withgesturing thresholds that include a gesturing threshold distance,usually as part of an initial slide.

7.6.3.2 Detection Process

The process of detecting a finger-mapped gesture entails detecting a setof gesturing fingers, determining their finger identities and othergesture parameters, and identifying the user request that corresponds tothe gesture parameters. The set of fingers that the gesture systemdetects gesturing is called the “gesturing finger set.” The set ofgesturing fingers actually used to determine finger identities is calledthe “selecting finger set”; at some point during the gesture, agesturing finger set becomes the selecting finger set. The set of fingeridentities that the gesture system associates with the fingers of theselecting finger set is called the “selected identity set.” Thisspecification often refers to finger identities of the selected identityset as just “finger identities,” when it is clear that the fingeridentities characterize a selecting finger set.

In its general form, the process of detecting a gesture includes thefollowing steps, as illustrated in FIG. 9, with some refinements shownin FIG. 10:

-   -   In step 120, the host or gesture system first prepares an input        area. If the input area is based on a hand calibration, the user        first provides this hand calibration, as in step 130 of FIG. 10.        If the host does not provide finger identities, the gesture        system uses the hand calibration to produce a finger map on the        input area, as in step 131 of FIG. 10.    -   In step 121, the gesture system monitors the input area for        gesturing fingers. Some condition occurs and establishes the        gesturing fingers as the selecting finger set, completing the        gesture. This condition is normally the lifting of at least one        gesturing finger. That is, the gesture system tests for the        condition of any gesturing finger lifting, of some number of        gesturing fingers having lifted, or of all of the gesturing        fingers having lifted. There may also be circumstances when the        gesture system can assign the selecting finger set before any        finger lifts. For example, the gesture system may monitor        gesturing fingers for their parameter values and assign the        selecting finger set once the fingers indicate a gesture        parameterization that cannot be further changed by further        gesturing.    -   In step 122, the gesture system determines a selected identity        set for the fingers of the selecting finger set. Either the host        provides the finger identities to the gesture system or the        gesture system deduces them using a finger map. FIG. 10 depicts        the finger map approach as step 132 and is explained later. The        gesture system may or may not associate each finger of the        selecting finger set with a distinct finger identity, as only        the full selected identity set is relevant to determining the        user request.    -   In step 123, the gesture system determines values for the        gesture parameters input by the fingers of the selecting finger        set and attempts to determine a user request for these        parameters. The gesture parameters include the selected identity        set of step 122. The particular gesture parameterization may or        may not correspond to a user request, however. In some gesture        systems, all parameterizations may be valid, such as in systems        that report the gesture parameterization itself to the host.        Other gesture systems may have to perform a look-up to associate        a user request with the parameterization, and some        parameterizations may not have associated user requests.        Depending on the gesture system, the process of determining        values for the gesture parameters may be indistinguishable from        the process of determining a user request.    -   In step 124, the gesture system reports the user request to the        host if the gesture parameterization is associated with a valid        user request.    -   In step 125, if any fingers of the selecting finger set remain        touching, the gesture system waits for them to lift before        looping back to step 121 to receive another gesture.

Subsequent sections of this specification express a variety ofrefinements of this general process of detecting gestures for differentpossible features of the gesture system.

7.6.3.3 Real-World Handling

The process of detecting a finger-mapped gesture should provide handlingfor intermittent real-world conditions. State-transition diagrams arebetter able to economically express such complex state handling thantraditional flow diagrams. FIG. 11 depicts a state-transition diagramthat inputs gestures and handles out-of-sequence conditions that canarise during the process of inputting gestures. The diagram depictsgeneral states of detecting a gesture and events that transition amongthese states, implementing an approach that identifies gesturing fingersas those that exceed a gesturing threshold and that detects gesturecompletion as the first lifting of a gesturing finger. See the section“State-Transition Diagrams” at the end of the accompanying sampleimplementation for an explanation of how to read the state-transitiondiagrams.

The transition events of the diagram in FIG. 11 are as follows:

-   -   140—If a finger begins gesturing, initialize a gesturing finger        set with the finger and transition to state 137.    -   141—If another finger begins gesturing, add the finger to the        gesturing finger set and remain in the current state.    -   142—If multiple fingers are gesturing and one of them lifts,        transition to state 138.    -   143—If multiple fingers remain gesturing and one of them lifts,        remain in state 138.    -   144—If the last gesturing finger lifts, report the gesture        outcome and transition to state 136.    -   145—If unrecognized gesturing movement is detected, transition        to state 139.    -   146—If excessive additional gesturing is detected, or if another        finger begins gesturing, transition to state 139.    -   147—If the last gesturing finger lifts, transition to state 136.

The states of the diagram in FIG. 11 are as follows:

-   -   Awaiting Gesture 136—This is the initial state of a gesture        session, in which no fingers that are touching the gesture        surface are considered to be gesturing, except possibly for        fingers that have moved while serving as “locked holds,” as        explained later in the specification. When a finger exceeds a        gesturing threshold, the session begins collecting a gesturing        finger set, which initially contains only this first finger, and        it transitions via transition 140 to the Selecting Gesture state        137.    -   Selecting Gesture 137—This is the first of two states 137-138 in        the Performing Gesture superstate 135. It is the state during        which the user selects the fingers that compose the gesture.        During this state, the gesture system collects the gesturing        finger set via transition 141; whenever another finger exceeds        the gesturing threshold, it is added to the gesturing finger        set, unless the finger is serving as a “hold,” as explained        later. Fingers can touch the gesture surface in any order and        exceed the gesturing threshold in any order. On the first        occurrence of a gesturing finger lifting from the gesture        surface, the session transitions out of this state via        transition 142 or 144, at which point the session establishes        the gesture's selecting finger set. The selecting finger set is        here the set of fingers in the gesturing finger set the first        time one of the fingers of the gesturing finger set lifts, with        the set including the lifting finger itself. If there is more        than one finger in the selecting finger set, the session        transitions via transition 142 to the Completing Gesture state        138; otherwise the session transitions via transition 144,        according to the Performing Gesture superstate 135.    -   Completing Gesture 138—This is the second state in the        Performing Gesture superstate 135. In this state, the user has        lifted at least one finger to end the gesture, and the session        is waiting for the user to lift the remaining fingers of the        gesture. That is, during this state, the session is waiting for        the user to lift all of the fingers of the selecting finger set.        Should the gesture system detect unexpected movement, either the        user did not intend the gesture to be interpreted as a        finger-mapped gesture or the user performed the gesture        incorrectly. Hence, while in the state, the session should        monitor the remaining fingers of the selecting finger set for        excessive additional movement and to transition to the Aborting        Gesture state 139 via transition 146 if detected. The session        also transitions to the Aborting Gesture state 139 via        transition 146 if it detects that another finger has begun        gesturing, because this amounts to the user apparently intending        to add another finger to the selecting finger set after having        indicated that there are no more fingers in the selecting finger        set. Otherwise, the session remains in this state until it        transitions according to its Performing Gesture superstate 135.    -   Performing Gesture 135—This is a superstate that contains the        Selecting Gesture and Completing Gesture states 137-138. The        session will transition via transition 145 out of either of        these two states into the Aborting Gesture state 139 if it ever        detects that one of the fingers in the gesturing finger set is        moving in a way not known for finger-mapped gestures. For        example, if the gesture system limits finger-mapped gestures to        lockstep gestures, as described later, the session transitions        to the Aborting Gesture state 139 if it ever determines that the        gesture in progress is not a lockstep gesture. However, if one        of the fingers of the gesturing finger set lifts and either        there are no other fingers in the gesturing finger set or all        other fingers of the gesturing finger set have already lifted,        the gesture has completed, and transition 144 occurs. In this        case, a selecting finger set will have been established, as        already explained, so the session translates the selecting        finger set into a gesture outcome and reports the gesture        outcome to the host. After reporting the gesture outcome, the        session returns to the Awaiting Gesture state 136 to await the        next gesture.    -   Aborting Gesture 139—In this state, the user is in the process        of performing a gesture that the gesture system does not        recognize. The session is waiting for the user to lift all        fingers that are gesturing but not serving as “locked holds,” as        described later. When the last such gesturing finger lifts, the        session transitions via transition 147 back to the Awaiting        Gesture state 136 to wait for the user to begin inputting a new        gesture. The gesture system may optionally report to the host        that the gesture is not a recognized finger-mapped gesture,        doing so either when the session transitions into the Aborting        Gesture 139 state or when the session transitions from the        Aborting Gesture 139 state to the Awaiting Gesture state 136.

FIG. 11 depicts one of many possible designs of a gesture session. Thisparticular design ignores all finger activity that it does notexplicitly handle. In particular, fingers may touch and lift from thegesture surface without affecting the gesture, provided that the fingersdo not exceed the gesturing threshold while they are touching. Thegesture system may also support “holds,” as explained later, that mustalso lift before the first gesturing finger lifts in order not to beinterpreted as performing a hold. This facility for ignoring accidentaltouching is helpful because the close proximity of multiple fingersrapidly performing finger-mapped gestures virtually guarantees that somefingers will briefly touch the gesture surface by accident.

When the host is also capable of receiving non-finger-mapped gestures onan input area, the gesture system should be careful to ensure that theuser indeed performed a finger-mapped gesture. The process depicted inFIG. 11 accomplishes this by waiting for the last finger of thegesturing finger set to lift before reporting the gesture outcome. Thisway the gesture system can avoid reporting a gesture outcome if thegesture ultimately ends up deviating from the expectations offinger-mapped gestures. This approach also has the benefit of allowingthe host to provide feedback indicating gesture completion, so the usercan see when the gesture system is ready for another gesture. However,it is also an option for the gesture system to report the gestureoutcome at the time the first gesturing finger lifts, when it hasdetermined the selecting finger set.

Under a variation of the approach depicted in FIG. 11, the gesturesession maintains a gesturing finger set of fingers that have gesturedand touched the gesture surface within the recent past. The sessiondelays determining the selecting finger set until the last gesturingfinger has lifted. Fingers are retained in the gesturing finger setafter they have lifted until they are older than a certain amount oftime. The selecting finger set is defined as the set of fingers thatremain in the gesturing finger set at the time the last gesturing fingerlifts. With an appropriately chosen amount of time, this approach helpsto eliminate accidental gesturing that occurs early in the gesture. Theapproach also allows the user to pause a gesture before completing toguarantee that only certain fingers occur in the selecting finger setand contribute to the gesture.

7.6.3.4 Example Gestures

A few examples help to illustrate the process of detecting gesturesusing gesturing thresholds. FIGS. 16 and 17 depict the same two-fingeredgesture. Vector 215 is the initial slide vector for finger A, and vector217 is the initial slide vector for finger B. Finger A lifts first atpoint 204. Finger B is at point 209 at this time. Fingers A and B arethe only gesturing fingers, so the selecting finger set consists offingers A and B. The gesture system can determine a gesture level and adiscrete direction at this time from the lengths and directions ofvectors 216 and 218. Meanwhile, the session is in the Completing Gesturestate 138 (FIG. 11) and finger B continues to gesture before finallylifting at point 211. If finger B travels too far beyond point 209 ortakes too much additional time to lift, the gesture system can declineto recognize the gesture. Finally, when finger B lifts at point 211, thegesture system reports the gesture outcome determined at points 204 and209 to the host.

FIGS. 18A-18H illustrate a two-fingered finger-mapped gesture. Each ofthese figures depicts an input area. A solid dot (e.g. point 230 in FIG.18A) represents a location where a finger is touching. An open dot (e.g.point 230 in FIG. 18B) represents a location that a finger previouslytouched; points change from solid to open during the course of thegesture. An arrow connecting dots indicates a finger's direction oftravel. An arrow with a dashed line (e.g. from point 230 to 231 in FIG.18B) indicates that the finger had not yet exceeded the gesturingthreshold for that leg of the path. An arrow with a solid line (e.g.from point 233 to 236 in FIG. 18E) indicates a leg of the path where thefinger is considered to be gesturing. A cross line through a point (e.g.through point 233 in FIG. 18C) indicates that the gesturing thresholdwas exceeded at that point. The capital letters A and B represent fingeridentities. The series of figures depicts two fingers traveling paths.The figures are further described as follows:

-   -   1) FIG. 18A. A finger with finger identity A has touched the        input area at point 230. The figure does not indicate how the        finger identity was determined for the finger. It could have        been determined according to its location on the input area. It        could also have been determined by the device from physical        properties that the device monitors.    -   2) FIG. 18B. Finger A moved from point 230 to point 231, but the        movement was not fast enough or not far enough to exceed the        gesturing threshold. Meanwhile, another finger with identity B        touched at point 232. Neither finger is gesturing because        neither has exceeded the gesturing threshold yet. Were either        finger to lift now, no finger-mapped gesture would be        registered.    -   3) FIG. 18C. Finger B remains motionless as finger A continues        to gesture. Finger A exceeds the gesturing threshold upon        reaching point 233: it is now gesturing and inputting a        finger-mapped gesture. The finger identities contributing to a        finger-mapped gesture are not determined in this example until        at least one of the gesturing fingers lifts, so finger B can        still participate in this gesture.    -   4) FIG. 18D. Finger A stops moving, but finger B begins moving        and exceeds the gesturing threshold on the very first segment of        the path at point 234. The gesture is now a two-fingered        finger-mapped gesture with finger identities A and B. Because no        gesturing finger has lifted yet, additional fingers may yet        contribute to the gesture.    -   5) FIG. 18E. Fingers A and B are now both moving, having moved        to points 236 and 235, respectively. This figure depicts the        predominant condition during two-fingered gesturing.    -   6) FIG. 18F. Finger B has lifted from point 235. No more fingers        can contribute to this gesture, so the gesture's selecting        finger set is now known to contain fingers A and B. The gesture        system can determine the indicated user request at this point.        The directions and distances that the fingers have so far        travelled can be used to determine a discrete direction and a        gesture level. However, it is best for the gesture system to        wait for the user to lift all of the gesturing fingers before        reporting the user request to the host.    -   7) FIG. 18G. By moving to point 237, finger A continues to        gesture after finger B has lifted. This is likely to happen        whenever humans try to simultaneously move two fingers at once        and lift them at once; one finger is likely to gesture for        slightly longer than the other.    -   8) FIG. 18H. Finally, finger A lifts too. The gesture is        complete. This is the ideal time for the gesture system to        report the user request to the host.

7.6.4 Gesture Types

It is useful for the gesture system to restrict the kinds of gesturesthat are interpreted as finger-mapped gestures beyond the minimalrequirements of finger-mapped gesturing. Each kind of gesture is calleda “gesture type.” Restricting gesture types allows both the host and thegesture system to monitor for gestures on the same input area bydetecting different gesture types: the host ignores gesture types thatthe gesture system recognizes, and the gesture system ignores gesturetypes that the host recognizes. Restricting the gesture types that thegesture system recognizes also trains the user to only input therecognized gestures types, and by making the recognized gesture typesthose that are most efficient for finger-mapped gesturing, therestrictions help train the user to be efficient. Restricting therecognized gesture types also helps to reduce erroneous input by havingthe gesture system ignore unusual input that might have been accidental,making the user still more efficient by reducing the number ofunintentional user requests the user would have to undo.

7.6.4.1 Lockstep Gestures

The most efficient gesture type for finger-mapped gesturing is thelockstep gesture. A “lockstep gesture” is a gesture in which allgesturing fingers move roughly the same distance in roughly the samedirection. It is an action that the user performs. The user performs alockstep gesture by deciding which fingers to include in the gesture andintending to move all of these fingers together as a unit, all in thesame direction for the same distance. Lockstep gestures are natural forthe human hand, as people readily move multiple fingers together whengesturing, whereas people require a lot of practice and skill tosimultaneously move multiple fingers independently. In practice, thefingers are likely to move in slightly different directions for slightlydifferent distances, and it is up to the gesture system to determinewhether these differences are within the tolerances of a lockstepgesture. The gesture system need only test gestures for being lockstepto disambiguate them from non-lockstep gestures. However, in order to beable to assign a direction to a gesture involving multiple gesturingfingers, the angles of the final distance vectors of the gestured pathsshould not span a range of more than π/2 radians for some classequivalent set of angles.

7.6.4.2 Stroke and Sweep Gestures

A lockstep gesture involving only one gesturing finger is called a“stroke,” and one involving two or more gesturing fingers is called a“sweep.” A gesture system may limit itself to recognizing only lockstepgestures involving a certain numbers of fingers, such as only strokes oronly two-fingered sweeps. More specifically, a “stroke” is a lockstepgesture in which the user places a single finger on the input area,moves the finger to a new location, and lifts the finger; it is also afinger-mapped gesture whose selecting finger set contains exactly onefinger. A “sweep” is a lockstep gesture in which the user places two ormore fingers on the input area, moves the fingers in roughly the samedirection for a distance, and lifts the fingers; it is also afinger-mapped gesture whose selecting finger set contains two or morefingers. FIGS. 14A-14H depict hands demonstrating two- andthree-fingered sweeps in various directions. Extended fingers (e.g.185-186) indicate fingers that are touching and gesturing, and arrows(e.g. 187-188) indicate direction of movement.

A simple approach to testing for movement roughly in the same directionis to require that the direction angles of every two fingers in theselecting finger set differ by at most a maximum amount by classequivalence. For example, π/2 radians is a reasonable maximum, allowingfingers to be considered travelling in the same direction if they alltravel in the same π/2 radian angle arc from their respectivemeasurement origins. FIGS. 16 and 17 illustrate this approach for atwo-fingered sweep. Both figures depict the same gesture. Finger A liftsat point 204 when finger B is at point 209, corresponding to distancevectors 216 and 218. The gesture system ensures that the angles ϕ_(A2)and ϕ_(B2) of these two distance vectors are roughly the same byrequiring that they differ by less than the maximum amount γ₁ by classequivalence. Alternatively, the gesture system can assign a discretedirection to each finger separately, for some directional partitioning,and require that the fingers all have the same discrete direction. Forexample, the angles ϕ_(A2) and ϕ_(B2) would both be required to indicatethe same discrete direction on an oriented partitioning, possibly adifferent oriented partitioning for each finger. Although π/2 radians isa reasonable γ₁ for ensuring that the gesture system can assign adirection to the gesture, a sweep typically consists of adjacent fingersof a hand, and adjacent fingers typically gesture along roughly parallellines. The dotted lines 180 represent the horizontal relative to whichdirections are determined.

It is less important that all of the fingers of a sweep move roughly thesame distance, because the user can indicate the distance intended forthe gesture as the average of the distance measures of all the fingersor as the greatest distance measure of the fingers, controlling theintended distance during the gesture. A lockstep gesture therefore neednot restrict all fingers to moving roughly the same distance. However, auser may more quickly learn to gesture effectively if the fingers arerequired to move approximately the same distance. A simple way to dothis is to require that no distance measure differ from any otherdistance measure by more than a certain percentage. However, a betterapproach accommodates the likelihood that different fingers tend to movedifferent distances when gesturing despite intending to indicate thesame gesture level. As explained later, this approach assigns a gesturelevel integer to each finger of the gesturing finger set and requiresthat none of the resulting gesture level integers differ by more than 1.

The gesture system of a multi-faceted touch device should select a meansfor determining directions that is comparable across facets. If thegesture system first maps all facets to a common gesture surface, therelative orientation of the facets on the gesture surface affectswhether fingers moving on different facets are moving in the samedirection. If the gesture system is designed with knowledge of thefacets, it can map the facets to appropriately oriented portions of thegesture surface. The obvious alternative is for the gesture system todetermine directions for each facet separately using the facet's owncoordinate system; remember that the gesture surface is only a devicefor making the language of the specification generic. Anotheralternative is for the discrete directions to be determined for eachfinger separately, according to the forward direction associated witheach finger, and to compare the directions of fingers by comparing theirdiscrete directions. Still another alternative is to compare finger“forward-relative” directions, as explained later.

Once the gesture system has determined that it has a lockstep gesture,it can report a gesture outcome under any of the following conditions:upon detecting a subsequent newly gesturing finger, upon detecting thefirst lifting of a gesturing finger, upon detecting that all fingers ofthe selecting finger set have lifted, or upon determining that nofurther modification to the gesture parameterization is possible. It ispossible for the gesture system to decide that it has a stroke upondetecting a subsequent newly gesturing finger, and to report a gestureoutcome at this time, but doing so precludes the gesture system fromalso detecting sweeps, because the subsequent gesturing finger may bethe second finger of a sweep. A gesture system that accepts both strokesand sweeps should therefore wait for at least one gesturing finger tolift and a selecting finger set to be defined before reporting a gestureoutcome. However, it is ideal to only report the gesture outcome afterall fingers of the selecting finger set have lifted, thus allowing theuser to interpret the host having performed a behavior as a signal thatthe gesture system is ready to receive another gesture.

7.6.4.3 Direction-Constrained Gestures

It can be helpful to require that lockstep gestures be performed in aconstant direction as the user performs the gesture, for the duration ofthe gesture. This allows the host to give special meaning to gestures inwhich the fingers change direction mid-gesture. This optional constraintforces users to perform lockstep gestures efficiently and unambiguously,which is particularly important when the gesture system includes agesture direction as part of a gesture parameterization. Lockstepgestures that are constrained in this way are called“direction-constrained” gestures. Here are some ways to require thatusers perform direction-constrained gestures:

-   -   1) Record the angle of the distance vector for each finger at        the moment the finger is detected to have travelled at least a        minimum distance, according to each finger's distance measure.        The gesture system subsequently monitors either the local angle        or the distance vector angle of each finger. The gesture system        fails to recognize the gesture if the angle monitored for a        finger ever subsequently differs too much by class equivalence        from the angle recorded for the finger. A maximum allowed        difference of π/4 is reasonable.    -   2) Assign each finger a discrete direction the moment the finger        is detected to have travelled at least a minimum distance,        according to each fingers' distance measure. The gesture system        subsequently monitors either the local angle or the distance        vector angle of each finger. If the discrete direction        determined for a monitored angle is ever found to disagree with        the finger's originally assigned discrete direction, the gesture        system fails to recognize the gesture. Consider the stroke shown        in FIG. 13. Vector 176 is the initial slide vector and has a        length at least the minimum initial slide length. A directional        partitioning 179 is superimposed centered on the measurement        origin 167. To constrain the direction, the gesture system can        require that the angle of the distance vector (e.g. angles ϕ₂        and ϕ₃ for distance vectors 177-178) always indicate the        discrete direction that the angle ϕ₁ of the initial slide vector        indicated. The dotted line 180 indicates the horizontal axis        relative to which direction angles are determined. All of the        distance vectors 176-178 have sampling angles in the FORWARD        range, indicating that the stroke has constant direction.    -   3) An approach only available to sweeps records a direction        angle for the sweep once all of the fingers have each travelled        the minimum distance. The direction angle for the sweep can be        computed, among other possibilities, as the nearest average of        the distance vector angles of all the fingers or as the nearest        bisection of the range of angles that the fingers' distance        vector angles span. The gesture system subsequently monitors the        local angles or the angles of the distance vectors for each        finger to compute a current direction angle for the sweep. The        gesture system fails to recognize a gesture if the current        direction angle ever differs too much by class equivalence from        the direction angle originally recorded for the sweep. Consider        the sweep shown in FIG. 16. Vectors 215 and 217 are the initial        distance vectors for fingers A and B, respectively, having        angles ϕ_(A1) and ϕ_(B1). Let ϕ_(AB1) be the nearest average of        these angles, establishing an initial angle for the sweep.        Vectors 216 and 218 represent distance vectors for the two        fingers at the same time. Let ϕ_(AB2) be the nearest average of        their angles ϕ_(A2) and ϕ_(B2). The gesture system can constrain        the direction by ensuring that ϕ_(AB1) and ϕ_(AB2) differ by        less than γ₂ by class equivalence as ϕ_(AB2) varies.    -   4) Another approach only available to sweeps records a discrete        direction for the sweep once all of the fingers have each        travelled the minimum distance. The discrete direction for the        sweep can be determined, among other possibilities, by using a        direction angle for the sweep as the sampling angle. The gesture        system subsequently monitors the direction angle for the sweep        to compute a current discrete direction for the sweep. The        gesture system fails to recognize a gesture if the current        discrete direction ever differs from the discrete direction        originally recorded for the sweep.

7.6.4.4 Non-Lockstep Gestures

Lockstep gestures are not the only possible finger-mapped gestures,though they are probably the ones most suitable for accurate gesturingat high speed. It is also possible to allow gestures such as switches,pinches, and zooms to be finger-mapped gestures. A “switch” is a gesturein which two gesturing fingers approach and pass each other beforelifting, essentially switching sides. For example, an index finger mightgesture backward as the middle finger gestures forward, passing eachother and performing a switch. A “pinch” is a gesture in which twogesturing fingers approach each other and lift without passing eachother. A “zoom” is a gesture in which two gesturing fingers move inroughly opposite directions before lifting. All of these can beinterpreted as finger-mapped gestures by parameterizing the fingeridentities and at least one property of the paths, such as the distancesor directions that the fingers travel.

A gesture system that supports multiple gesture types for finger-mappedgestures can further parameterize gestures as a function of gesturetype. For example, when a two-fingered sweep gesture and a switchgesture are performed one after the other, they could use the same twofinger identities and have the same gesture level but indicate differentgesture outcomes. Strokes and sweeps are distinguishable by the numberof finger identities, so by virtue of already selecting gesture outcomeas a function of finger identities, a gesture system need not furtherdistinguish between strokes and sweeps. That is, a gesture system thatonly supports lockstep gestures need not further parameterize gesturesaccording to whether they are a strokes or sweeps.

7.6.5 Region Conflict Resolution

On touch devices that do not directly indicate finger identities,gesture systems infer finger identities from the input regions touched.During a finger-mapped gesture, the gesture system associates eachfinger of the gesturing finger set with an input region. Each finger isnormally associated with its originating region. However, when inputtingcombinations of fingers on facets that contain multiple input regions,it is possible for multiple fingers of the gesturing finger set to havethe same originating region. Unless the situation is resolved, thegesture system would as sign both fingers the same finger identity,yielding a conflict. The gesture system can handle the conflict bytreating the gesture as unrecognized, but doing so may frustrate theuser, who likely only accidentally originated the fingers on the sameregion. To make gesturing more responsive to the user, the gesturesystem can instead determine the regions that the user most likely meantto touch and re-associate the fingers with these regions, thusreassigning finger identities. The gesture system resolves regionconflicts using the order in which the regions occur in the finger map.

The point of time at which a gesture system must perform region conflictresolution depends on the gesture system. In order to allow the gesturesystem to monitor fingers according to the requirements of theirassociated regions, the gesture system should perform conflictresolution for each newly touching finger by the time the finger isadded to the gesturing finger set, according to the process previouslydescribed for determining the gesturing finger set. For example, as partof constraining the direction of sweeps, the gesture system may monitorthe discrete direction of each finger in the gesturing finger set,computing the discrete direction using a reference direction that varieswith the finger's associated region. This requires that the gesturesystem reference a finger's properly associated region when the fingerbecomes part of the sweep's gesturing finger set. Hence, if a gesturesystem that enforces sweep direction in this manner is also to performregion conflict resolution, conflict resolution for a newly touchingfinger must be performed by the time the finger is determined to be inthe gesturing finger set. However, if the gesture system is notmonitoring fingers according to the requirements of their associatedregions, the gesture system need not determine finger identities untilit determines the gesture outcome.

There are two conflict conditions that the gesture system can handle ina straightforward way. First, the gesture system need not determine theidentities of fingers that do not gesture, unless the fingers turn outto be performing holds (which are explained later). This allows multiplefingers to be touching the same region, potentially creating ambiguity,but resolving ambiguity by giving precedence to gesturing fingers. Ifone finger is holding and the gesture system needs its identity, thegesture system can apply the normal means for resolving conflictsbetween two fingers. Second, if as many fingers are touching as thereare regions, the gesture system can simply have the selecting finger setindicate all of the finger identities. If it is important to have eachfinger assigned to a region, such as for purposes of confirming alockstep gesture, the gesture system can arbitrarily re-assign one ofthe fingers of the conflicting region to the input region to which nofinger is assigned. Less straightforwardly, for other conflictconditions, when two fingers originate on the same region, the gesturesystem can reasonably assume that the finger that is nearest to anotherregion actually belongs to that other region. The gesture system appliesheuristics for determining which finger is nearer to another region andre-associates the finger with that region.

A general way to uniquely associate fingers with their nearest regionsis to associate each finger with a distinct region in a way thatminimizes the sum of the distances from each finger's initial touchlocation to the finger's prospective associated region. The distancefrom a finger's initial touch location to a prospective region could bethe distance from the initial touch location to the location in theprospective region that is nearest to the finger's initial touchlocation. Although this generic rule works no matter how many fingersparticipate in the gesture, in practice a finger map is based on a handcalibration, and if three or more fingers ever have the same originatingregion, it is most likely that either the hand calibration is for a userother than the user who is gesturing or the user has positioned the handon the finger map entirely incorrectly. So when the gesture system findsthree or more fingers of the gesturing finger set to have the sameoriginating region, it is best for the gesture system to treat thegesture as unrecognized. It is only pragmatic to provide conflictresolution when only two fingers have the same originating region.

When two fingers are found to have the same originating region, they canbe re-associated with regions according to one of two rules. The rulesemploy the distance between an initial touch location and a region. Thedistance between an initial touch location and a region other than theone that contains the initial touch location is the length of the linesegment that starts at the initial touch location and ends at thelocation in the adjacent region that is nearest to the initial touchlocation. For regions defined by a hand calibration, this can be thedistance between the initial touch location and the nearest point on theline that forms the nearest boundary of the region. The rule applieddepends on the situation. The rules are as follows:

-   -   1) The first rule applies when the common originating region is        adjacent to only one other region; that is, it applies when the        common originating region is at one end of a series of regions.        In this case, the finger whose initial touch location is closest        to the adjacent region is re-associated with the adjacent        region, and the other finger retains its original region        association. If the two initial touch locations are equidistant        from the adjacent region, the gesture system may arbitrarily        re-associate one of the fingers with the adjacent region or it        may treat the gesture as unrecognized for its remainder. Ideally        however, the gesture system would postpone the re-association        until the first time at which one finger is found to occur        closer than the other finger to the boundary of the adjacent        region. When the gesture system depends on knowing the        association, this latter tactic requires delaying interpreting        the finger as a member of the gesturing finger set until the        re-association is made.    -   2) The second rule applies when the common originating region is        adjacent to two distinct other regions. In this case, the        gesture system first determines the midpoint between the two        initial touch locations. The midpoint is either closer to the        one of the adjacent regions or it is equidistant from both        adjacent regions. If it is closer to one of the regions, the        fingers are re-associated according to the previous rule as if        the only two regions under consideration were the region        containing the midpoint and the adjacent region that is closest        to the midpoint. If the midpoint is equidistant from both        adjacent regions, the gesture system cannot reliably        re-associate either finger unless one of the adjacent regions is        already associated with a finger, in which case one of the        fingers may be arbitrarily associated with the other adjacent        region. If the gesture system cannot perform a re-association,        the gesture is treated as unrecognized for its remainder.        Ideally however, the gesture system would postpone the        re-association until the first time at which the midpoint is        calculated to be closer to one of the adjacent regions. When the        gesture system depends on knowing the association, this tactic        requires delaying interpreting the finger as a member of the        gesturing finger set until the re-association is made.

FIGS. 20A and 20B depict two scenarios in which two fingers have initialtouch locations in the same region, region 282, of a finger map havingfour regions 280-283. In FIG. 20A the fingers initially touch locations284 and 285, and in FIG. 20B the fingers initially touch locations 287and 288. If the fingers gesture before they lift, the gesture systemre-associates one of the fingers with a different region. Each scenariohas a location 286 or 289 marked at the midpoint between the two initialtouch locations. The gesture system determines the distance from thismidpoint to each of the adjacent regions 281 or 283. The distance from amidpoint to region 281 is given as U_(L), and the distance from amidpoint to region 283 is given as U_(R). In these scenarios, thegesture system is measuring the distance U_(L) or U_(R) from themidpoint 286 or 289 to the adjacent regions as the length of the linesegment having one terminus on the midpoint and another terminus on theregion boundary 290 or 291, such that the line segment is perpendicularto the region boundary. This is the distance from the midpoint 286 or289 to the nearest location on the boundary 290 or 291. In the firstscenario, FIG. 20A, the midpoint 286 is closer to region 283, so thefinger to the right of the midpoint 286 having initial touch location285 is re-associated with region 283. In the second scenario, FIG. 20B,the midpoint 289 is closer to region 281, so the finger to the left ofthe midpoint 289 having initial touch location 287 is re-associated withregion 281. The gesture system can also use this technique to determinethe selected identity set when a conflict arises, without individuallyre-associating the fingers.

In a gesture system that requires all finger-mapped gestures to belockstep gestures, the user is unlikely to be able to sense which of thetwo fingers gets re-associated with the identified adjacent region. Thisis because the user intends the fingers of the lockstep gesture to movein the same direction for the same distance. A gesture system maytherefore arbitrarily re-associate one of the fingers with theidentified region. However, there is a chance that this re-associationshortcut might cause the gesture to be interpreted incorrectly if thegesture system includes gesture direction as part of a gestureparameterization and the user's fingers tend to move apart during asweep gesture, provided that the finger map reflects this tendency ofthe user's hand.

Upon re-associating a finger with a new region, the finger's identity isreassigned according to its new region. If one of these rules is appliedwhile the gesturing finger set includes three or more fingers, it ispossible for the region re-association to conflict with an alreadyestablished association of a finger with a region. This conflict canoccur when the fingers shift too far from home position on the fingermap. In this case, it is reasonable and perhaps helpful for the gesturesystem to treat the gesture as unrecognized until it completes, invitingthe user to correct back to home position or establish a new handcalibration. Alternatively, the gesture system can shift eachconflicting finger association over by one, only treating the gesture asunrecognized if the last finger requiring re-association cannot bere-associated because there is no next adjacent region.

FIG. 19 depicts an incorporation of conflict resolution into the stepsfor processing a finger-mapped gesture. FIG. 19 is expressed in terms ofa generalization of input regions that has not been discussed yet. Tointerpret this figure for input regions, interpret a “geometric element”as an input region, and interpret the “nearest” geometric element to afinger as the input region that contains the finger's initial touchlocation. The process for determining the selected identity set has beenexpanded into several steps 270-273, the first of which 270 determinesthe input regions, referred to here as “geometric elements,” that arenearest to the fingers of the selecting finger set. Test 274 checkswhether two fingers have an initial touch location in the same inputregion. If each finger of the gesture has an initial touch location in adifferent input region, the selected identity set includes the fingeridentities to which the each of these different input regionscorresponds, as shown in step 273. If two fingers of the gesture have aninitial touch location in the same input region, the selected identityset includes the finger identity to which that input region correspondsand also the finger identity to which one of the adjacent input regionscorresponds, as shown in step 271. The choice of adjacent input regioncan be made using one of the techniques previously described. If therewas a conflict of two fingers in a gesture of three or more fingers,step 272 includes in the selected identity set the finger identities towhich correspond the input regions in which each of the remainingfingers has an initial touch location. At this point, the gesture systemhas the complete selected identity set and can proceed with determiningthe user request in step 123.

7.6.6 Retraction and Revocation

A gesture system may optionally implement retraction. “Retraction” is aprocess whereby the user moves the fingers to perform a lockstepgesture, tracing paths for the gesture, but prior to lifting thefingers, roughly retraces the fingers in reverse along their paths forat least a portion of their paths. The process of moving roughly in areverse fashion reduces the distance measures reported for the fingers,which in turn may change the value of the gesture level or of othergesture parameters, hence also changing the outcome that would beselected for the gesture were the fingers to lift at the retractedlocation. Retraction is useful for providing users with assistancelearning the gesture system. As the user moves the fingers to perform agesture, the gesture system provides the host with status noticesindicating the outcome that the gesture would have if the fingers wereto lift at their current locations. The host can provide this real-timefeedback to the user.

The gesture system may also support revocation. A “revoked” gesture is agesture that completes with a special gesture outcome called a“revocation.” A gesture is in a revoked state when any of its gesturingfingers acquires a distance measure less than or equal to a revocationthreshold length, such as the gesturing threshold distance. A revocationproduces no effect except perhaps the sending of a status notice to thehost indicating its occurrence. Revocation allows the user to beginperforming a gesture, but to decide against completing the gesture, andto do so in a way that isn't reported as either a user request or a usererror. Revocation is most useful to people who use retraction to explorethe set of possible outcomes available to a gesture prior to selectingone, as when training to use the gesture system. Revocation provides aneasy way for these people to simply drop the gesture without effect sothey can check another gesture.

Because users may forget where their fingers began gesturing, it's notpragmatic to base revocation detection exclusively on the proximities offingers to their measurement origins. Revocation is best employed with adistance measure determined by projection onto lines parallel to eachfinger's initial direction of travel, allowing the user to revoke agesture by retracting each finger into a half-plane corresponding todistance measures equal to or below a specified revocation thresholdvalue. Distance measures in the direction opposite the measurementorigin from a finger's initial direction of travel can be treated asnegative lengths.

FIG. 21 illustrates the revocation of a stroke gesture. The finger hasan initial touch location at point 307 and exceeds a gesturing thresholdat point 309. Exceeding the gesturing threshold establishes the initialslide vector 306 and the measurement origin 308. The gesture systemdetermines a discrete direction for the gesture using the initial slidevector 306 and a quarter-plane directional partitioning 305 relative tothe measurement origin 308. A projection line 304 is establishedbisecting the quarter plane 305 that contains the initial slide vector306 and intersecting the measurement origin 308. The “P=” notationsindicate the distance measure at each point, where the distance measureis the length of the distance vector when projected onto the projectionline 304. The figure illustrates distances 300-303 at increments of 10.The measurement origin 308 is at distance 0. The finger continuestravelling and reverses direction at point 310, which projects onto theprojection line 304 at distance 30. The finger continues its reversal ofdirection until it reaches point 312. Suppose the gesture systemconsiders projection distances of 10 or less to be revocation. In thiscase, lifting the finger at point 312 revokes the gesture. Were thefinger to instead lift at any preceding point, the distance measure ofthe gesture would be the value of P at the point.

The gesture system can take advantage of negative distance measures toprovide an additional feature called “re-origination.” Re-originationeffectively pushes the measurement origin back to the farthest pointthat the finger ever reaches in the negative-value half-plane. Underthis approach, a finger's effective distance measure never drops belowzero, and every movement again in the original direction of the gestureincreases the value of the measure. Re-origination is easily implementedwith a gesture level by calculating the gesture level as a function ofeach finger's distance measure minus the most-negative value everachieved by the distance measure. Re-origination makes it easy for theuser to revoke a gesture by pulling fingers backward any distancegreater than the forward distance they travelled, while also allowingthe user to move subsequently forward again as if the finger werestarting a new gesture at the point at which it reversed direction.

Retraction, revocation, and re-origination are primarily helpful fortutorial or instructional purposes. They are not necessary because it isjust as easy—if not easier—for the user to complete a mistaken gesture,undo the mistake if the gesture corresponded to a known behavior, andthen perform the correct gesture. This is the same reason that aconventional computer keyboard need not provide a way for a user to undohaving pressed the letter ‘A’ before the user lifts the keyboard buttonfor ‘A’; the user can always lift the mistaken key press, delete the‘A’, and then type the actually intended letter. In fact, most keyboardsregister the key pressed at the time the key is pressed rather than atthe time the key is lifted, eliminating the possibility of revocation.

7.7 Parameter Types

A finger-mapped gesture includes a selection of finger identities and atleast one additional gesture parameter. This section describes some ofthe possible gesture parameters, focusing on those that a user can inputwithout having to change the hand position during the gesture. Gestureoutcomes are a function of the parameter values indicated by thesegesture parameters, in addition to being a function of the selectedfinger identities. This specification defines the following gestureparameters: gesture level, touch count, holds, gesture direction,backbrushing, and multi-reversal. A finger-mapped gesture system canprovide the user with easy access to a diversity of user requests bysupporting multiple gesture parameters, thus enabling a great variety ofparameterizations. Gesture level and gesture direction are the mostbasic gesture parameters, because they only require movement for adistance, which is the minimal action of a finger-mapped gesture.

7.7.1 Gesture Level

The gesture level of a finger-mapped gesture is an abstract measure ofthe length of the gesture that is dependent on at least one lengthproperty of the gesture. When a user inputs a gesture level as part of agesture, the user selects a gesture outcome according to how far thefingers travel during the gesture. Gesture levels partition possiblelengths into ranges, where each range corresponds to a distinct gesturelevel. The length partitioning allows the user to input varying gesturelengths that all select the same gesture level. The gesture systemtranslates one or more length measures of a gesture to a gesture level.Because the user strives to input a single length when selecting agesture level, finger-mapped gestures that include a gesture levelparameter are best treated as lockstep gestures.

7.7.1.1 Determining a Gesture Length

Many different length properties may characterize a gesture. A length ofa gesture depends on distances that the fingers travel along theirgestured paths. In particular, a length depends on the gesturingfingers' measurement origins and level selection points. The “levelselection point” of a finger is the point on the finger's path that isused for determining the gesture length relative to a measurementorigin. The gesture system can define a finger's level selection pointas any of the following or even as the first of any of the following tooccur:

-   -   1) The point at which the finger lifts.    -   2) The point on the path at which the finger had its maximum        distance measure during the gesture.    -   3) The point at the time the selecting finger set is determined.

The distance measure of a finger at the level selection point is calledthe finger's “selected distance.” The length of a stroke gesture is thefinger's selected distance. The length of a sweep can be a function ofthe selected distances of the gesturing fingers or more generally afunction of the fingers' measurement origins and level selection points.Techniques for determining a gesture length for a sweep gesture includethe following:

-   -   1) The gesture length is the selected distance of just one of        the fingers of the selecting finger set, such as the selected        distance of the leftmost or the rightmost gesturing finger.    -   2) The gesture length is the average of the selected distances        of the fingers. FIG. 16 illustrates this approach with a        two-fingered sweep. Finger A lifts at point 204 while finger B        is still touching at point 209. If the level selection points        are determined at the time of the selecting finger set is        determined, both points 204 and 209 are level selection points.        The selected distances are therefore the lengths of the distance        vectors 216 and 218. Under this approach, the gesture length is        the average of the lengths of vectors 216 and 218.    -   4) The gesture length is the length of the interpolated distance        vector of the fingers. The “interpolated distance vector” is the        vector that starts at the average measurement origin and ends at        the average level selection point. The average measurement        origin has a location given by separately averaging the x- and        y-values of the measurement origins of the fingers in the        selecting finger set. Similarly, the average level selection        point has a location given by separately averaging the x- and        y-values of the level selection points of these fingers. The        result is a single virtual measurement origin and a single        virtual level selection point. The gesture length can be the        distance between these two virtual points.

Figures in this specification preferentially illustrate gesture lengthand discrete direction being determined for sweeps using an interpolateddistance vector because the interpolated distance vector is readilyamenable to graphical representation. FIG. 17 depicts the details ofcomputing an interpolated distance vector for a two-fingered sweep.Finger A has a measurement origin at point 200 and finger B has ameasurement origin at point 205. Point 212 is the average measurementorigin. Finger A lifts at point 204 while finger B is still touching atpoint 209. If the level selection points are determined at the time ofthe selecting finger set is determined, and if the selecting finger setis determined at the time that the first gesturing finger lifts, bothpoints 204 and 209 are level selection points. Point 214 is the averageof these points and hence an average level selection point. The gesturelength in this example is therefore the length of vector 220, whichstarts at the average measurement origin 212 and ends at the averagelevel selection point 214. Because different facets may employ differentcoordinate systems for finger locations, interpolated distance vectorscannot necessarily be calculated between fingers that are on differentfacets. The direction of a sweep that is evaluated as an interpolateddistance vector can be constrained to the direction detected for thesweep when both fingers of the sweep have exceeded a gesturingthreshold. As illustrated in FIG. 17, the varying direction indicated bythe angle ϕ_(AB4) can be constrained to the direction originallyindicated by the angle ϕ_(AB3) interpolated at the gesturing thresholdof both fingers.

7.7.1.2 Determining the Gesture Level

The gesture lengths available to a gesture can be partitioned intoranges of length. A “gesture level” represents a particular range ofgesture lengths corresponding to a value of a gesture level parameter.The form of this representation within the gesture system is immaterial.The gesture system could directly represent a gesture level value as arange of lengths, but it is convenient to represent the value with anidentifier. For example, gesture levels could be labeled as letters ofthe alphabet, as the names ‘first’, ‘second’, ‘third’, etc., or with thewords “short”, “medium”, “long”, and “extra long.” This specificationindicates each gesture level with an integer. Each integer correspondsto a different range of gesture length, with increasing values of thegesture level integer corresponding to ranges of increasing length. Theminimum length for the range of a particular gesture level is called the“threshold distance” or “threshold length” for that gesture level.Different fingers may employ different threshold lengths, as maydifferent combinations of fingers. For convenience of discussion, assumethat in the context of a particular set of threshold lengths, the lowestselectable range of lengths has gesture level 1, that the next higherrange has level 2, and so on. Integer levels are useful because they canbe averaged together or compared numerically. To be clear that theaccompanying claims are not restricted to the representation of gesturelevels as integers, the claims may refer to the lowest gesture levellength range as the “first-range” gesture level and the second lowestlength range as the “second-range” gesture level.

The gesture level of a gesture is dependent on the range of lengths inwhich each relevant length of the gesture falls; determining the gesturelevel of a gesture is a matter of determining these ranges of lengths.This is straightforward for a stroke gesture, because the finger'sselected distance is the only length available: the gesture level of astroke is determined from the finger's selected distance. There areseveral ways to assign a gesture level to a sweep gesture, becauseseveral length computations are available. One approach is to determinea single gesture length for the entire gesture and determine a gesturelevel from this length. Any of the gesture length computations describedin the previous section for a sweep serve this purpose. It is alsopossible to determine multiple intermediate gesture lengths and anintermediate gesture level for each intermediate gesture length, andthen to determine a gesture level for the entire gesture fromintermediate gesture levels. The following methods exemplify this lattertechnique:

-   -   1) The gesture level for the gesture is the average of        intermediate gesture level integers determined separately for        each of the fingers. Under this approach, the gesture system        first maps each finger's selected distance to the gesture level        integer that the gesture would have were that finger the only        member of the selecting finger set. The result is one integer        per finger of the actual selecting finger set. The gesture level        integer for the gesture as a whole is a rounded average of these        integers, indicating a particular gesture level. This approach        enables the user to train each finger to independently learn the        lengths of its gesture levels and to control the gesture level        of a multi-finger sweep gesture in terms of these learned        per-finger gesture levels. The difference between this approach        and averaging the selected distances of all the fingers is        subtle, but it may be significant for adept users.    -   2) The gesture level integer for the gesture is the greatest of        intermediate gesture level integers determined separately for        each of the fingers. Under this approach, the gesture system        first maps each finger's selected distance to the gesture level        integer that the gesture would have were that finger the only        member of the selecting finger set. The result is one integer        per finger of the actual selecting finger set. The gesture level        integer for the gesture as a whole is greatest of these        integers. This approach allows the user to benefit from        independently training fingers to their associated ranges of        gesture levels. It is also forgiving of fingers that happen to        be a little lazy while gesturing. The gesture system may        optionally decline to recognize the gesture if any two        intermediate gesture level integers differ by more than one,        helping to keep the gesture from being ambiguous. If the        selecting finger set contains three or more fingers, the gesture        system may limit the number of fingers allowed to have gesture        level integers different from the maximum, which may help to        reduce the chance of misinterpreting the gesture. This approach        also offers the best method for ensuring that all fingers of a        sweep move roughly the same distance, as is ideal in a lockstep        gesture.    -   3) The gesture level integer for the gesture is the smallest of        intermediate gesture level integers determined separately for        each of the fingers, analogous to the previous example.

The gesture system may employ a combination of different methods fortranslating gestures to gesture levels. In particular, a function thattranslates a gesture length into a gesture level may vary by otherparameters of the gesture. Consider that different fingers tend togesture different lengths, indicating that it would be useful todetermine gesture level thresholds separately for each finger, perhapsbasing each threshold on the lengths of channel line segments determinedfor the fingers. It may also be useful to vary gesture thresholds by thedirection in which the fingers move. Consider that a user may beinclined to gesture fingers farther when moving them forward than whenmoving them backward; the gesture level computation would ideallyaccommodate a user's individual inclinations. The gesture system mayalso employ any of the possible distance measures, even in combinations.However, at least for flat touch-sensitive surfaces, using a strictlylinear distance between a measurement origin and a sampling point iseffective.

7.7.1.3 Gesture Level Examples

FIG. 23 illustrates multiple possible gestured paths for each of twofingers identified as A and B. ‘A’ and ‘B’ each appear next to thestarting location, 370 or 381 respectively, of a branching tree ofpaths. Each branching tree of paths, or “path tree,” depicts multiplepaths that a single finger may travel as part of a gesture, with eachbranch indicating where one possible path deviates from another. Thestarting location, 370 or 381, represents a measurement origin for eachof the associated finger's possible paths. The labels ‘A’ and ‘B’ alsoidentify their associated path trees.

Each path tree of FIG. 23 has three legs, 402-404 or 405-407, connectingnumbered points 370-388. The first leg, 402 or 405, is shown as a solidline, the second leg, 403 or 406, is a dashed line, and the third leg,404 or 407, is a dotted line that originates midway down the dashed leg403 or 406. Only the first leg, 402 or 405 respectively for fingers A orB, is relevant to the present discussion; the second and third legsdepict “backbrushing,” which is described later. Arrows represent thedirection of travel of a finger. The solid arrow 400 or 401 originatingat each measurement origin, 370 or 381, represents an initial slidevector. Three arcs, 389-391 or 394-396, partition the first leg of eachfinger's path. Each arc represents a distance from the measurementorigin 370 or 381 corresponding to a minimum distance for a gesturelevel. In addition to being labeled with detail numbers, each arc islabeled with a parenthesized letter that represents the distance of thearc from its associated measurement origin.

The tables of FIGS. 24A-24C indicate how to map gesture level distancesto user requests for purposes of the example of FIG. 23. The userrequests in this case are unparenthesized letters and ampersand. Thedashed entries in the tables indicate conditions that cannot arise. Eachrow of a table represents a gesture level, indicated to the left of therow as “GL” followed by a gesture level integer. Each column of a tablerepresents a “backbrush level,” which for purposes of the presentexample is 0 (“BL 0”), meaning that the gesture does not “backbrush.”“Backbrushing” and “backbrush level” are described later. The tablesassume that gesture level and backbrush level are the only gestureparameters. Table 24A applies to a gesture involving only finger A,table 24B applies to a gesture involving only finger B, and 24C appliesto a sweep involving both fingers A and B. Consider the followingscenarios:

-   -   1) Finger A gestures alone and lifts at point 375. The distance        from point 370 to point 375 exceeds distance (c) of arc 391.        Cross-referencing “>(c)” and “BL 0” in the table in FIG. 24A        indicates that this gesture has gesture level 3 (“GL 3”) and        inputs the character ‘G’.    -   2) Finger B gestures alone and lifts at point 384. The distance        from point 381 to point 384 is less than distance (h) of arc 396        but greater than distance (g) of arc 395. The table in FIG. 24B        indicates that this gesture has gesture level 2 (“GL 2”) and        inputs the character ‘M’.    -   3) Fingers A and B together perform a sweep gesture, lifting at        points 375 and 384, respectively. If the gesture system assigns        the gesture level of a sweep to the greatest gesture level of        its gesturing fingers, then finger A provides the gesture level        for the sweep, because finger A gestured at gesture level 3,        while finger B only gestured at gesture level 2. The table in        FIG. 24C therefore indicates that the gesture inputs the        character ‘Y’.    -   4) Fingers A and B together perform a sweep gesture, lifting at        points 373 and 384, respectively. Each finger indicates a        gesture level integer of 2, so the gesture level for the gesture        is 2. The table in FIG. 24C indicates that this gesture inputs        the character ‘V’.    -   5) Finger A gestures alone and lifts at point 371. This is also        the point at which the gesture system detected an initial slide,        which must have a minimum length. Because there is a minimum        distance from the measurement origin that a finger can gesture,        there is also a minimum distance for the first gesture level. If        point 371 is at the minimum initial slide length, distance (a)        is the lowest possible gesture distance. The tables 24A-24C        designate distances below this as gesture level zero, showing        dashes to indicate that it is not possible to select gesture        level 0. This gesture occurred at distance (a), so the table in        FIG. 24A indicates that it inputs the character ‘A’.    -   6) Finger B gestures alone and lifts at point 382. Distance (f)        represents the minimum initial slide length, but the gesture        system did not detect the initial slide until a greater distance        away, at point 382. Distance (f) is also the threshold distance        for gesture level 1, so this scenario is a level 1 gesture.        Table 24B indicates that this gesture inputs the character ‘J’.

7.7.2 Gesture Direction

A gesture may be parameterized by the direction in which the userperforms the gesture. A “gesture direction” parameter is a gestureparameter that is a function of a direction property of the gesture. Inparticular, a gesture direction is a function of the range of directionsin which a direction property of the gesture falls, becausefinger-mapped gestures distinguish among only a limited number ofclasses of direction. Each range of directions is called a “discretedirection,” as described for directional partitionings. It is best thata gesture for which a gesture direction is being determined be lockstep,so that the gesture system may reliably assign a discrete direction tothe gesture as a whole. This section describes some techniques that areavailable for determining a gesture's discrete direction. The techniquesare described with direction angles representing directions, but theycan also be applied to other representations of directions. According toa generic process, the gesture system determines a “gesture angle” forthe gesture as a whole and then determines a discrete direction from thegesture angle. The discrete direction is read from a directionalpartitioning oriented to a reference angle that is suitable for thegesture, where the reference angle represents a previously establishedstandard direction, such as a center-forward direction that makes sensefor the gesturing fingers. The discrete direction parameterizes thegesture outcome.

7.7.2.1 Determining Direction Angles

The gesture system determines the discrete direction of a gesture fromat least one direction property of the gestured paths of the gesture.The generic process described here determines one or more directionangles representing one or more directions of the gesture and thendetermines a discrete direction from the direction angle or directionangles. A direction angle can be determined separately for each fingeror collectively for a group of fingers.

There are many ways to assign a direction to each finger, but thesimplest is to use the direction angle of one of the distance vectors ofthe finger. The first useful distance vector is that of the point atwhich the finger is found to exceed a gesturing threshold, assuming thepresence of a gesturing threshold. The last useful distance vector isthat of the point at which the finger lifts from the gesture surface.The angles of any of these distance vectors suffice, but the gesturesystem should be consistent with the points relative to which itdetermines the distance vector. When determining a direction angle for agroup of gesturing fingers, the gesture system can use a vector thatoriginates at a midpoint between starting points for the fingers andthat ends at a midpoint between ending points of the fingers. FIGS. 16and 17 illustrate determining direction angles for individual fingersand for the gesture as a whole, respectively.

If the gesture system constrains the directions of lockstep gestures,the direction angle assigned to a finger can be the angle of thefinger's distance vector at nearly any time, including the time at whichthe finger lifts. A gesture system that supports multiple facets indifferent coordinate systems should assign “forward-relative” directionangles to the fingers for use in determining the gesture's discretedirection, unless the gesture system knows the relative orientations ofthe facets. Forward-relative angles are discussed later in thespecification.

In practice it appears that it is best to wait for a finger to travel abit beyond the distance at which it exceeds the gesturing thresholdbefore determining the finger's direction angle. That is, the gesturesystem should determine the direction angle of a finger after thefinger's distance measure equals or exceeds a minimum distance that isgreater than the gesturing threshold distance, for reasonable gesturingthresholds. This distance is the “direction assignment distance.” Agesture system can employ a direction assignment distance even in theabsence of a gesturing threshold. Because the user may end a gesture atany point after the fingers have begun gesturing, the gesture system mayhave to assign a direction angle to a finger before it would normally doso. In particular, if the gesture system normally waits for a finger totravel a direction assignment distance before assigning its directionangle, and if the finger lifts before travelling this distance, thegesture system should determine a direction angle relative to the pointwhere the finger lifted.

7.7.2.2 Determining the Gesture Angle

The gesture system most straightforwardly determines a discretedirection for a gesture from a single direction that it associates withthe gesture. An angle representation of this single direction is calledthe “gesture angle.” The gesture system computes the gesture angle fromthe direction angles that it determined for the gesture. The gestureangle of a stroke is normally the direction angle of the finger thatperformed the stroke. There are numerous ways to compute the gestureangle of a sweep.

One way to determine the gesture angle of a sweep is to use the nearestaverage of the fingers' direction angles, as illustrated in FIG. 16 fora two-finger gesture. The nearest average may optionally be weighted bythe distance measures of the fingers. However, the most flexiblyadaptive approach is to set the gesture angle to the nearest bisectionof the fingers' range direction angles. For example, if the gesture is atwo-fingered sweep and one finger has an angle of 40 degrees and theother an angle of 80 degrees, the gesture angle would be 60 degrees,which also happens to be the nearest average of the angles. If thegesture is a three-fingered sweep and the finger angles are 40, 43, and80 degrees, the range of angles is 40 to 80 degrees, so the nearestbisecting angle is again 60 degrees, which this time differs from thenearest average.

Another option is to have the gesture angle be the angle of a vectorlocated midway between where the fingers start gesturing and where thefingers complete the gesture. The interpolated distance vector is anexample of this for two fingers, as illustrated in FIG. 17. For anarbitrary number of fingers, the starting location of the midway vectorcould be the average of the measurement origins of the fingers, and theending location of the midway vector could be the average of the finallocations of the fingers. It is also an option to simply set the gestureangle to the direction angle of one of the fingers, but if the gesturesystem calibrates for the different directions in which differentfingers tend to move, it generally is better to infer the gesture anglefrom the overall direction of the gesture.

It is possible to implement the gesture system so that it determines adiscrete direction for the gesture by reducing the paths of the fingersto one of a set of predetermined gesture angles. This can be done byprojecting the path of each finger onto a previously establishedreference line associated with the finger and identifying a directionangle for the finger according to the direction along the path that theresulting projected vector points. The projected vector starts at thepoint on the reference line at which the measurement origin projects andpoints in the direction of the end of the path. The direction angle ofeach finger is one of the two direction angles of the reference line.Each finger may also be associated with multiple reference linesintersecting at a point. In this case, the gesture system associates thefinger with the reference line having a direction angle closest to thedirection angle of finger's path. The gesture system then projects thepath onto that reference line and establishes a direction angle for thefinger equal to the angle of the projected vector. In the case of asweep gesture, it is also possible to project a virtual pathrepresenting the gesture onto a reference line and directly reduce themulti-path gesture to one of the angles of the reference line. Theseprojection approaches allow gesturing fingers to indicate directionsalong predefined axes akin to the directions available to a fingerscrolling an image on a touchscreen.

7.7.2.3 Determining the Reference Angle

In order for the gesture system to assign a discrete direction to agesture, it must first establish which direction is forward for thegesture. It is possible to establish a forward direction for the gestureas a whole because the fingers move forward in roughly the samedirection. Establishing the gesture's forward direction allows thebackward direction and other directions to be established relative toit. One way to determine these ranges of directions is to use agesture's forward direction as the reference angle for orienting adirectional partitioning. The directional partitioning is oriented sothat its center-forward direction is aligned with the reference angle,also orienting the discrete directions for the gesture. The value of thegesture direction parameter would then be the discrete direction thatcontains the gesture angle. Under another approach, the gesture systemdetermines a discrete direction by projecting a distance vector onto areference line having angles that indicate opposing directions ofmovement, such as onto a reference line representing forward/backwardmovement. In this case, the discrete direction is given by the directionin which the projected vector points along the reference line. Thisapproach also has a “reference angle,” defined as the angle for forwardmovement along the forward/backward reference line. Regardless of theapproach implemented, the direction of forward movement must first beestablished. This section describes methods for establishing thisdirection, expressing the direction as a reference angle.

FIGS. 27A-27D illustrate some of the possible reference angles that agesture system may use. In these figures, the points 450-453 representrest locations, the lines 462-465 represent forward directions for thegesture, the angles θ₁, θ₂, θ₃, and θ₄ represent reference angles, andthe angles θ_(A), θ_(B), and θ_(D), represent forward angles forfingers. The simplest reference angles are constant, as in FIG. 27A. Forexample, the reference angle θ₁ can be specific to the host or the touchdevice and hence constant across gestures. A constant reference angle isreasonable for devices that constrain the user's hand to a particularposition and orientation, such as a smartphone with a small screen. FIG.27B shows an example of a computed reference angle. The reference angleθ₂ is the angle perpendicular to the orientation line 458 of a handcalibration, with the angle θ₂ pointing in the general direction thatdifferent finger forward angles would point. Under this approach, thereference angle is constant for a particular finger map, which is usefulwhen finger maps can vary in orientation. FIG. 27C shows anotherreference angle θ₃ that is constant for a finger map. θ₃ is the nearestbisection of the range spanned by forward angles for all of thenon-thumb fingers, which is the bisection of the angles θ_(A) and θ_(D)at the ends of the range. FIG. 27C shows the range as spanning an arclength of A. This latter approach defines the forward direction of agesture as a function of the forward directions of the individualfingers.

The gesture's reference angle represents the forward direction of thegesture, so it is most effective to establish this forward direction asa function of the forward directions of the fingers that perform thegesture. If the forward directions are all identical, the referenceangle is the angle of the forward direction. Otherwise, a handcalibration can provide the finger forward directions as forward anglesfor each finger identity. In this case, when the gesture is a stroke,the reference angle of the gesture is just the forward angle of thefinger that performed the stroke. When the gesture is a sweep, thevarious ways for determining the gesture angle as a function of fingerdirection angles may also be employed for determining the gesture'sreference angle as a function of the finger forward angles. For example,the reference angle can be the nearest average of the forward angles ofthe fingers, and this average can optionally be weighted by the fingers'distance measures. FIG. 27D illustrates the averaging of forward anglesθ_(A), θ_(B), and θ_(D) for three fingers, yielding the reference angleθ₄. The dashed lines 459-461 represent the channel lines for thesefingers. Analogously to the best approach for determining the gestureangle, however, it is probably most helpful to the user for the gesturesystem to set the gesture's reference angle to the nearest bisection ofthe range of angles spanned by all forward angles for the fingers of theselecting finger set.

On multi-faceted devices whose facets use different coordinate systems,the forward directions of different fingers may not be directlycomparable. Recall that the convention in this specification is for thedirection angle to be determined relative to the positive x-axis; thex-axes of different facets may not physically coincide on the touchdevice. In this scenario, it may not make sense to compute the gesture'sreference angle as the average of finger forward angles. If the relativeorientations of the facet x-axes are known, the forward directions canbe averaged by first adjusting them according to the orientations of thex-axes. If the relative orientations of the facet are not known, such asif they are configured in ways not communicated to the gesture system,the gesture system can instead compute a “forward-relative” directionangle for each finger and then average the forward-relative angles toproduce an angle for the gesture overall, as explained below.

7.7.2.4 Determining the Discrete Direction

The discrete direction value of a gesture direction parameter can bedetermined from the gesture's direction, the direction of forwardmovement, and the relative orientations of the various discretedirections to the direction of forward movement. According to thegeneric process being described here, once both the gesture angle andthe reference angle are known, the gesture's discrete direction can bedetermined from a directional partitioning. The directional partitioningcontains at least the gesture angles that are possible for the user toinput. If the touch device interprets gestures for any directionavailable for gesturing, the directional partitioning contains allpossible direction angles. If the touch device treats some directions asinvalid, the gesture system must distinguish valid directions frominvalid directions, so the invalid directions can be considered tobelong to a discrete direction that designates them as invalid. Sometouch devices may restrict the movement of the finger or limit thedirections of movement that can be detected, in which case the gesturesystem can get away with a directional partitioning that contains only afew directions.

The choice of directional partitioning depends on the requirements ofthe gesture system. The half-plane partitioning shown in FIG. 25A issufficient for a gesture system that only recognizes forward- andbackward-moving gestures. The quarter-plane partitioning shown in FIG.25C may be the most prudent for recognizing four distinct directions,because it gives the user the maximum possible room for error whenselecting a direction, such that any further apparent error in adirection is reasonably interpreted as an intentional selection of adifferent discrete direction. A gesture system may opt for a variant ofthe bow-tie partitioning of FIG. 25B or a variant of the half-symmetricfour-way partitioning of FIG. 25D to require that the user employ carewith selecting certain discrete directions, in order to reduce thechance that these directions will be selected by accident. These lattertwo directional partitionings may be useful for reserving certaindiscrete directions for more consequential user requests. The gesturesystem can employ the same directional partitioning in every gesture,perhaps orienting it for each gesture, but it is also an option to varythe directional partitioning as a function of the identities of thegesturing fingers.

Technically, the gesture system could decline to recognize gestureshaving certain discrete directions, for example by requiring that allgestures move forward or backward and that none move left or right.Similarly, the gesture system could opt to only detect two discretedirections and require that all valid gestures be performed in just oneof them, such as requiring that all directions be in the BACKWARD rangedespite also detecting directions in the FORWARD range. However, thevalue of the limitations is unclear, except possibly to force the userinto a gesturing pattern that minimizes user error. Hence, the parametervalues available to a gesture direction parameter normally include eachdiscrete direction that the gesture system recognizes.

FIG. 13 illustrates determining the discrete direction of a strokeaccording to an oriented partitioning. The oriented partitioning has acenter-forward angle of approximately π/2. The finger lifts at point 175at distance vector 178. Both the initial slide vector 176 and thedistance vector 178 indicate a discrete direction of FORWARD, soregardless of the gesture angle ϕ₁, ϕ₂, or ϕ₃ that the gesture systemuses, this is the discrete direction of the gesture. The discretedirection determined would be the same even if the gesture were to endwith a “backbrush,” which is described later.

FIGS. 16 and 17 illustrate the determination of sampling angles foridentifying the discrete direction of a two-fingered sweep. The paths ofthe fingers in FIG. 16 are identical to the paths of the fingers in FIG.17. Finger A lifts at point 204. Finger B is still touching at this timeat point 209. The distance vectors for points 204 and 209 are 216 and218, respectively. In FIG. 16, the sampling angle is calculated asϕ_(AB2), which is the nearest average of the angles of the distancevectors 216 and 218. FIG. 17 depicts interpolated distance vectors.Point 212 is the midpoint between points 200 and 205, point 213 is themidpoint between points 201 and 207, and point 214 is the midpointbetween points 204 and 209. In FIG. 17, the sampling angle is given asangle ϕ_(AB4), which is the angle of the interpolated distance vector220 at the time finger A lifts. This figure does not depict the processof determining the oriented partitioning with which to compare thesampling angle. FIGS. 27A-27D show several ways that a reference anglecan be chosen for orienting a directional partitioning. The directionalpartitioning to orient can be one that the gesture system universallyapplies across all fingers, it can be a directional partitioningassociated with one of the fingers labeled A and B in FIGS. 16 and 17,or it can be a directional partitioning derived from those individuallyassociated with fingers A and B. The discrete direction determined wouldbe the same even if the gesture were to end with a “backbrush,” which isdescribed later.

It is possible to implement a gesture direction parameter so that it isa function of multiple discrete directions determined for a gesture frommultiple gesture angles. For example, the gesture system can determine adiscrete direction separately for each finger and then average thediscrete directions together, selecting a discrete direction for thegesture as the discrete direction nearest to this average. Representingdiscrete directions as integers facilitates this approach.Alternatively, the gesture system can select one of multiple computeddiscrete directions without combining them. When computing discretedirections for each finger, it is useful to constrain the differences indirections allowed among the fingers, such as to enforce lockstepgestures. For example, the gesture system may require that all fingersindicate the same discrete direction.

Finally, consider a gesture system that projects paths onto referencelines for purposes of determining the discrete direction of a gesture.This reduces each path to one of a set of predefined gesture angles.Each gesture angle corresponds to a discrete direction, so thedirectional partitioning of the generic process need only contain thesefew predefined gesture angles. The direction of the vector that resultsfrom the projection identifies the discrete direction. For example, ifthe gesture system hard codes the FORWARD and BACKWARD discretedirections to represent directions along the y-axis, the direction ofthe gesture can identify the discrete direction simply according towhether the y-value at the end of a path is greater than the y-value ofthe start of the path. As this example makes clear, it is not necessaryto represent directions as angles, it is not necessary to representdiscrete directions as ranges of multiple directions, and it is notnecessary to represent a collection of discrete directions in anystructure resembling directional partitionings.

7.7.2.5 Forward-Relative Directions

When the user performs a sweep with fingers on multiple facets and therelative orientations of the facets are not known, the gesture systemshould determine gesture direction from the “forward-relative” directionof the fingers. A “forward-relative” direction is a direction determinedfor a finger relative to a forward direction that is associated with thefinger. In particular, a generic process expressed in terms of anglesdetermines a forward-relative angle relative to a forward angle for thefinger. The forward-relative direction angle is computed in effect firstby rotating the facet's coordinate system so that a vector having thefinger's forward angle rotates to an arbitrary predetermined “alignmentangle.” It helps to visualize the alignment angle as π/2, placing theforward direction at π/2 radians. A forward-relative direction angle ofπ/2 then indicates a direction identical to that of the forward angle,regardless of what the forward angle happens to be for the fingerrelative to the coordinate system of the facet. In order to be able tocompare the forward-relative direction angles of different fingers, thealignment angle must be the same for all fingers. For example, π/2 wouldbe the forward direction for the forward-relative direction angles ofall fingers. The alignment angle serves as the gesture's referenceangle, although this alignment angle is only defined for comparison withthe finger forward-relative direction angles.

FIGS. 28A-28G illustrate a process for determining forward-relativedirection angles ϕ_(A0) and ϕ_(D0) of a two-finger sweep using analignment angle of π/2, thus setting the forward angle in the alignedsystem to π/2. FIG. 28A depicts a finger map with regions 470-471 on onefacet identifying fingers A and B, with regions 472-473 on another facetidentifying fingers C and D, and with the barrier 478 dividing the twofacets. The relative orientations of the facets are clear in the figure,but assume that the gesture system does not have this information. Thedashed lines 474-477 are channel lines for the regions 470-473. θ_(A)and θ_(D) are the forward angles for the forward directions of channellines 474 and 477, respectively. Vector 487 in FIGS. 28B-28C representsthe forward direction of channel line 474, and vector 488 in FIGS.28D-28E represents the forward direction of channel line 477.

FIG. 28A shows a finger gesturing in region 470 along points 479-481 andanother finger gesturing in region 473 along points 482-484. The gesturesystem identifies the former finger as finger A and the latter finger asfinger D. The figure depicts finger A's distance vector 485 along withthis distance vector's direction angle ϕ_(A) and length X as shown inFIG. 28B. FIG. 28D also depicts finger D's distance vector 486 alongwith this distance vector's direction angle ϕ_(D) and length Y. FIG. 28Bdepicts the direction angle ϕ_(A) of finger A relative to the forwardangle θ_(A) associated with the region 470 that it is gesturing in. FIG.28D depicts the direction angle ϕ_(D) of finger D relative to theforward angle θ_(D) associated with the region 473 that it is gesturingin. FIGS. 28B and 28D help with understanding how forward-relativedirection angles are determined.

FIG. 28C depicts the calculation of finger A's forward-relativedirection angle ϕ_(A0). ϕ_(A0) is the angle of finger A's distancevector 485 in a coordinate system for which the forward angle is π/2.FIG. 28C depicts this as rotating the x-axis until it is perpendicularwith the forward direction 487, placing the forward direction at angleπ/2, and then determining the direction angle ϕ_(A0) of the distancevector 485 in this rotated coordinate system. FIG. 28E illustrates thesame derivation for finger D, with finger D's forward-relative directionangle ϕ_(D0) being the angle of the distance vector 486 in a coordinatesystem rotated to be perpendicular to the forward direction 488. Noticethat the forward-relative direction angles ϕ_(D0) and ϕ_(D0) are bothrelative to a forward direction of π/2 in their respective rotatedcoordinate systems.

FIG. 28F plots both of the distance vectors 485 and 486 according totheir forward-relative direction angles ϕ_(A0) and ϕ_(D0) in the virtualcoordinate system that has a forward direction at π/2, illustrated asvector 489. Δ is the nearest difference between the two angles ϕ_(A0)and ϕ_(D0). The gesture system can use this difference to test thegesture for being lockstep. The line 490 bisects the span of the anglesϕ_(A0) and ϕ_(D0), passing through the open dot 491, which representsthe average end point of the vectors 485 and 486. FIG. 28G depicts thevector 492 of this average end point 491. Its angle ϕ is the average ofthe two forward-relative direction angles ϕ_(A0) and ϕ_(D0), and itslength Z is the average of the lengths of two fingers' distance vectors485 and 486, shown as lengths X and Y. As shown in FIG. 28G, the angle ϕis used as the gesture angle for assigning a discrete direction to thegesture. Here it indicates the discrete direction FORWARD on thedirectional partitioning 493. The gesture length Z can be used fordetermining a gesture level for the sweep.

7.7.3 Touch Count

A “touch count” is the number of times one or more fingers touch thesame spot in rapid succession before performing an action. The actionmay be a finger-mapped gesture, in which case the touch count furtherparameterizes the gesture. The action may also be a hold, as describedlater, further selecting the type of hold. The touch count of afinger-mapped gesture or a hold is a function of the touch counts of theindividual fingers. The process of inputting a touch count of two ormore is called “multi-tapping” because each touch prior to the last is atap. Touch counts are determined from the number of paths in the touchhistories of fingers rather than from properties of gestured paths,although the presence of a gestured path may contribute to a touchcount.

7.7.3.1 Multi-Tapping

To perform multi-tapping, each finger performing the gesture or holdinputs a touch count. The touch count of a finger is the number of timesthat the finger touches the same spot within time constraints. Everyfinger that is touching the gesture surface has a touch count. Thattouch count is at least 1, a value indicating that the finger is on itsfirst touch. For supported fingers, lifting and touching the same “spot”again increases this value by 1 with each lift and touch. Fingers thatmulti-tap with touch counts of 1, 2, and 3 are said to “single-touch,”“double-touch,” and “triple-touch,” respectively. A multi-tapping fingerhas multiple initial touch locations in its touch history, and althoughany of these may be used to determine the finger's identity, a user islikely to target the first initial touch location to a region withgreatest accuracy.

The touch count of a finger-mapped gesture is a function of the touchcounts of its gesturing fingers. Specifically, it is a function of thetouch counts of each finger of the gesture's selecting finger set. Eachfinger normally inputs the same touch count. If there is only onegesturing finger, the touch count of the gesture is that of the finger.If there are multiple gesturing fingers and they all input the sametouch count, the touch count of the gesture is this touch count. Ifthere are multiple gesturing fingers and their touch counts differ, thegesture system selects the touch count as a function of the touch countsof the individual fingers. For example, the gesture system can use theaverage touch count or the greatest touch count. The gesture system canalso decline to recognize the gesture when the touch counts differ. Thegesture's touch count further parameterizes the gesture and hence alsothe gesture outcome. As with the touch counts of fingers, gestures withtouch counts of 1, 2, and 3 are called “single-touch,” “double-touch,”and “triple-touch” gestures, respectively.

The gesture system ideally uses a gesturing threshold to distinguishtaps from gesturing. In this case, once a finger exceeds the gesturingthreshold, it can no longer increase its touch count. It is alsopossible to identify touches according to a non-gesturing condition,such as a by detecting a touch that lasts less than a maximum amount oftime, but this approach increases the odds that the gesture system willfail to distinguish taps from other gesture actions. Although a user mayinput any touch count by tapping any number of times, the gesture systemis most pragmatic using only touch counts of at most 2 or 3, as userswill likely have trouble remembering distinct meanings for arbitrarilylarge touch counts. To be pragmatic, the gesture system can either failto recognize gestures having fingers with touch counts greater than amaximum touch count or it can treat all touch counts greater than orequal to a maximum touch count as equivalent.

Consider an example of a user entering a stroke. A finger that touchesthe gesture surface and immediately performs the stroke selects a touchcount of 1 without having to do anything special; 1 is the minimum touchcount. To perform stroke with touch count 2, the finger taps a spot onthe input area, lifts, and quickly touches down again on the same spot,performing the gesture from there. To use a touch count of 3, the fingerquickly taps the same spot twice before quickly touching again tofinally perform the gesture. In general, to input a touch count of N, afinger must input N−1 distinct touches-and-lifts before touching againto begin gesturing.

7.7.3.2 Determining a Touch Count

In order to allow multi-tapping, the gesture system must record the spotthat a finger touches. Here are three possible ways to define the spot:

-   -   1) The spot is a circle centered on the finger's initial touch        location, having some pre-specified radius. The user increases        the touch count by touching a finger again in this spot. After        the second touch, the finger likely has an initial touch        location different from the initial touch location of its first        touch. The gesture system may continue to employ the original        spot for detecting subsequent touches, or it may establish a new        spot centered on each subsequent initial touch location,        allowing the finger to drift more with repeated touches.    -   2) The spot is the input region that contains the initial touch        location of the finger's first touch. To increase the touch        count, the user need only touch again on the same region. This        approach assists a user who is envisioning a lockstep gesture        while inputting a touch count. The finger or fingers will likely        be in motion for the lockstep gesture by the final touch of the        multi-tapping, placing at least the final touch at some distance        from the initial touch.    -   3) The spot is the entire input area. When the host provides        finger identities, the gesture system need not monitor for        subsequent touches according to previous touch locations,        because the gesture system already knows which finger is        touching. It is possible to use this approach when the host does        not provide finger identities, provided that the gesture system        does not support multi-touch sweep gestures, because the touch        count can apply to the next gesture to be performed, even if        several fingers contributed to the touch count. Using the input        area as the touch spot is likely incompatible with the presence        of user interface elements on the input area that are activated        by touching without gesturing.

On a finger map, the ideal solution combines the first two approaches,increasing the touch count either when a finger touches within a radiusof a previous touch or when a finger touches the same input regionagain. By itself, the first approach does not handle rapid data entrywell, and by itself, the second approach is prone to error when the useris touching close to a boundary between regions, where the user mightaccidentally select different regions on successive touches.

7.7.3.3 Delineating Multi-Tap Gestures

Multi-tapping complicates gesture detection because the lifting offingers normally delineates gestures. The following constraints allow agesture system to distinguish sequential gestures when supportingmulti-tapping:

-   -   1) A finger that exceeds a gesturing threshold can no longer        increase its touch count. This provides a clear delineation        between gestures, as otherwise the gesture system might merge        sequential gestures together by having a subsequent intended        gesture actually just increase the touch count of the previous        gesture.    -   2) If after a finger has touched and lifted, a finger of a        finger identity that is not registered as having previously        touched for the present gesture itself touches and lifts, odds        are that the new finger is beginning a new gesture, so best        practice is to assume that multi-tapping is no longer in        progress for the previous gesture. This forces the previous        gesture to complete and allows a subsequent gesture to begin.        This also forces the user to ensure that when performing a sweep        multi-tap, all tapping fingers must be simultaneously touching        before any tapping finger lifts, at least for the first tap of        the gesture.    -   3) A finger that has touched, lifted, and remained lifted for        more than a specified period of time can no longer increase its        touch count. This allows a user to abort a gesture simply by        failing to complete the gesture within a reasonable time. It        also enables the gesture system to ignore random non-gesturing        touches that the user did not intend to have meaning. The        specified period of time is called the “maximum multi-tap up        time.”

The gesture system should also implement the following constraint:

-   -   4) A finger that is touching the gesture surface for more than a        specified period of time can no longer increase the touch count.        Technically, there is no design conflict with subsequently        increasing the touch count, but enforcing a time limit on        individual touches may help the user to remember to abide by the        time limit that exists on lifts after touching.

Constraints (3) and (4) together impose a minimum rate at which the usermust perform multi-tapping to increase a touch count. Since most usersare familiar with double-clicking on a mouse or on a touchpad, whichgenerally also require tap input at a minimum rate, most users shouldfind it natural to have to multi-tap at a minimum rate.

FIGS. 29A-29H illustrate the process of detecting a multi-tap gestureand counting touches by combining a tap radius with a tap region,allowing the user to increase the touch count by tapping within eitherthe tap radius or the original region. Each figure depicts the samefinger map having three regions 500, 501, and 502, corresponding tofinger identities A, B, and C, respectively. A solid dot (e.g. point 503in FIG. 29A) represents a location where a finger is touching. An opendot (e.g. point 503 in FIG. 29B) represents a location that a fingerpreviously touched. All dots 503-508 are numbered in the order in whichtheir indicated locations are touched. An arrow connecting dots (e.g.from points 506 to 507 in FIG. 29H) indicates a finger's direction oftravel. The FIGS. 29A-29H depict a series of input events on the fingermap.

In FIG. 29A, a finger touches at point 503 in region 501, and in FIG.29B, the finger lifts. The dashed circle 509 indicates a radius aroundthe point 503 where the finger touched. In FIG. 29C, a finger touches atpoint 504 outside of the radius 509 but within the same region 501,doing so in less time than the maximum multi-tap up time. Because thefinger is in the same region 501, the touch count increases to 2,indicated by the single circle around the dot 504 in FIG. 29C. In thisparticular example, a new radius 510 is established around the new touchlocation 504. The finger lifts in FIG. 29D and touches again quicklyenough in FIG. 29E at point 505. This time it touches in region 500,outside the original region 501. However, because it is inside theprevious tap radius 510, the touch count increases again, now to 3,depicted with another concentric circle at point 505. The finger liftsagain in FIG. 29F, which depicts a radius 511 around point 505, and ittouches again in FIG. 29G at point 506. Point 506 is outside theprevious tap radius 511, but it does touch within the original region501, so the touch count increases yet again, now to 4, illustrated withanother concentric circle around point 506. As FIG. 29H depicts, thefinger subsequently remains touching and performs a gesture along points506-508. When the gesture completes, it will be further parameterizedwith a touch count of 4.

7.7.3.4 Multi-Finger Touch Counts

When multiple fingers contribute to the same finger-mapped gesture, thetouch count of the gesture itself is a function of the touch counts thatthe fingers of the selecting finger set have at the time the selectingfinger set is established. If all of the fingers have the same touchcount, the gesture itself is assigned this touch count. When the fingershave different touch counts, the touch count of the gesture should bethe highest of the touch counts. Selecting the highest touch countrequires that at least one finger sufficiently perform the gesture,whereas selecting the lowest touch count requires that all fingerssufficiently perform the gesture. It is best for the gesture system todecline to recognize a gesture if the touch counts differ by too much,because the more internally inconsistent the gesture, the more likelythe gesture system is to misidentify the gesture that the user intended.It is reasonable to allow touch counts to differ by at most one from themaximum touch count input and to treat the gesture as unrecognizedotherwise.

It is also possible to parameterize a finger-mapped gesture according tothe touch counts of the fingers separately for each finger identity. Forexample, given two sweeps having the same gesture level, a sweep havingan index finger at touch count 2 and a middle finger at touch count 1could yield a different gesture outcome than a sweep having an indexfinger at touch count 1 and a middle finger at touch count 2. However,it is may be unreasonable to ask the user to remember differentinterpretations for each possible combination of touch counts, so thisspecification does not treat this approach further.

FIGS. 30A-30H illustrates a multi-tap two-fingered sweep on a finger mapof three regions 515-517 identifying fingers A, B, and C. The notationused in these figures is the same as the notation used in FIGS. 29A-29H,with all dots 518-525 being numbered in the order in which theirindicated locations are touched. Finger B touches at point 518 in FIG.30A and lifts in FIG. 30B. Meanwhile in FIG. 30B, finger C touches atpoint 519 in region 517. The finger at point 519 remains down in FIG.30C as finger B touches again at point 520, increasing finger B's touchcount to 2. In FIG. 30D, finger B begins gesturing, travelling to point521, while finger C lifts for another touch at point 522 in FIG. 30E. InFIG. 30E, both fingers have the same touch count, but finger B has begungesturing and finger C has not. Even with finger B already gesturing,finger C lifts from point 522 in FIG. 30F to increase the touch countagain. Finger C touches again at point 524 in FIG. 30G for a touch countof 3. Finally in FIG. 30H, finger C begins gesturing. There are now twogesturing fingers, one with touch count 2 at point 523 and the otherwith touch count 3 at point 525. The fingers may lift in either order toend the gesture, but neither may lift again to further increase itstouch count. This gesture ends has a touch count of 3 if gesture systemassigns the whole gesture a touch count equal to the greater of the twotouch counts. FIGS. 30A-30H also demonstrate that fingers may increasetouch counts or otherwise join a gesture in progress up until theinstant that one of the gesturing fingers lifts. It is an option for thegesture system to disallow fingers from increasing touch counts aftergesturing has begun.

FIGS. 30A-30H can also be interpreted as representing two multi-tapstroke gestures performed at different times. For the first sequence,assume that the user only gestures in region 516 and region 517 remainsempty. In FIG. 30A-30C, the user touches a finger, lifts, and touchesagain, establishing a touch count of 2. In the remaining figures, theuser moves the finger to point 523. When the user lifts the finger, theuser will have input a stroke with touch count 2. For the secondsequence, assume that the user only gestures in region 517 and region516 remains empty. The user touches the finger at point 519 in FIG. 30B,lifts the finger in FIG. 30D, and touches again in FIG. 30E for a touchcount of 2. The user then lifts the finger again in FIG. 30F and touchesagain in FIG. 30G for a touch count of 3. In FIG. 30H, the user finallygestures the finger to point 525, indicating a stroke. Upon lifting thefinger, the gesture will be a stroke with touch count 3.

7.7.4 Holds

A “hold” is a placement of relatively stationary fingers on the inputarea in a way that permits them to stay on the input area concurrentlywith gesturing fingers. Each finger participating in a hold is a “holdfinger.” It is possible to have multiple simultaneous holds, each withits own set of relatively stationary fingers, each performing a distinctfunction. When the user lifts from the gesture surface any of thefingers participating in the hold, the hold ends—though perhaps after adelay—and the function of the hold ceases. Ending a hold is called“releasing” the hold. A user may “retain a hold” across multiplegestures, if desired, before releasing it. A finger that is performing ahold is not considered to be gesturing and is not inputting a gesturedpath.

7.7.4.1 Hold Types

Each hold functions as a shift, an anchor, or an escape. A “shift” is ahold that further parameterizes a finger-mapped gesture; the outcome ofa gesture employed with a shift is further a function of the particularshift performed. In effect, a shift selects an alternate set of gestureoutcomes the way a shift key selects an alternate set of characters on aconventional keyboard. Just as there can be multiple types of shiftbuttons on a conventional keyboard, such as the shift key, the functionkey, and the control key, a gesture system can support multiple types ofshift holds. The particular shift that applies is given at least in partby the identities of the fingers that are performing the shift. Theshift that applies may also depend on the touch counts of the holdfingers in gesture systems that support multi-tapping.

An “anchor” is a hold that the gesture system ignores; an anchor doesnot further parameterize a gesture. Anchors allow the user to keepfingers on the input area without them interfering with the gesture thatthe remaining fingers are performing. For example, a gesture system maytreat a finger identified as the thumb as an anchor and thus allow theuser to keep the thumb on the gesture surface to hold the hand steadywhile the remaining fingers gesture. Anchors holds are helpful forkeeping the hand in home position and aligned with the finger map,particularly when the thumb performs the anchor hold. An anchor hold“anchors” the hand to the finger map. Once identified as an anchor, ananchor finger may move while touching and even exceed the gesturingthreshold without influencing gesture outcomes.

An “escape” is a hold that suspends finger-mapped gesturing for itsduration and allows other kinds of gestures to be input instead. Thegesture system detects the hold and delegates input events to anothersystem of gesturing while still monitoring for the hold to lift. When itdetects the hold lifting, it ceases to interpret input events in theother system and resumes finger-mapped gesturing. The touch count of theescape hold finger can select the particular class of gestures in whichto interpret input events for the duration of the hold.

FIGS. 15A-15C depict hands performing holds. Extended fingers (e.g. 189and 190) indicate fingers that can gesture during the holds, notnecessarily all simultaneously. The arrows (e.g. 191 and 193)demonstrate directions in which gesturing fingers can move during theholds, although other directions may also be possible. The dottedcircles 194-195 around fingers indicate fingers that are performingholds. The hand in FIG. 15A is performing a hold with the thumb 194 andgesturing with the remaining fingers. This could easily be either ananchor hold or a shift hold. The hand in FIG. 15B is performing a holdwith the pinky finger 195 and gesturing with the remaining non-thumbfingers. The sample implementation included with this specificationemploys a pinky finger shift hold in this manner for inputting numbercharacters. The hand in FIG. 15C is performing a hold with the thumb 194and gesturing with the index finger, illustrating an escape hold. Thearrows 193 indicate that the index finger is potentially free to move inany direction.

7.7.4.2 Establishing a Hold

When a finger touches the gesture surface at a location where it couldbe inputting a gesture, the gesture system does not yet know whether thefinger represents a hold. A finger is identified as a hold under certainconditions, and before those conditions are met, the finger is onlypotentially a hold finger. A finger in this state is a “potential holdfinger.” A set of potential hold fingers represents a “potential hold”when the combination of their identities and touch counts is a validcombination for a particular type of hold, as established by the gesturesystem. It is an option for the gesture system to interpret any fingerthat touches a specific area of the gesture surface as performing ashift function, and it is an option for the gesture system to interpretany touch of a specific finger identity as performing a shift function,but these conditions preclude the finger from also possibly performing afinger-mapped gesture after touching. These fingers are not consideredto be performing holds. Only finger touches that have to bedisambiguated from finger-mapped gestures perform holds; hold fingersmust start out as potential holds.

There are a few possible conditions that a potential hold finger mustsatisfy to become a hold finger, depending on the gesture system. Underone approach, a potential hold finger can only become a hold finger ifit satisfies a non-gesturing condition by gesture completion. Thegesture system may or may not require that the hold finger remaintouching until gesture completion. Under another approach, a finger canonly become a hold finger if it has not exceeding a gesturing thresholdby the time the gesture completes. Because non-gesturing conditions canslow data entry, it is best to identify potential hold fingers asfingers that have not exceeded a gesturing threshold. This specificationtherefore generally assumes the use of gesturing thresholds with holdfingers.

Although the hold condition must be satisfied by the time the gesturecompletes, the gesture system may require that the hold fingers beestablished sooner. For example, the gesture system could interpret apotential hold as a hold at the time some finger exceeds the gesturingthreshold at the start of the gesture, although this approach providesthe user with no wiggle room for coinciding holds with initiatinggestures. It also does not allow the user to use the short time spentperforming the gesture to recall whether the gesture requires a hold andthus perform the hold before completing the gesture. In some gesturesystems, it may be possible for the user to indicate different holdsover the course of the gesture, requiring the gesture system to selectone of them. The gesture system could interpret the first simultaneousset of potential hold fingers to occur during the gesture as the holdfingers, or it could interpret the last simultaneous set as the holdfingers. If a hold can have at most one finger, then either the first orthe last potential hold finger of the gesture would become the holdfinger. If a hold can consist of multiple fingers, the gesture systemcould instead identify the largest number of simultaneously touchingpotential hold fingers as the hold fingers of the gesture, regardless ofwhether they formed the first or last hold of the gesture. However, ifthe hold fingers can input a touch count to further select the shiftmode, only the last potential hold fingers would become hold fingers,because previous touches would be increasing the touch count.

If any non-gesturing finger on the input area can indicate a hold, theuser may often input unrecognized gestures or incorrect user requests asfingers accidentally touch and accidentally indicate holds.Consequently, it is ideal that the gesture system use the potential holdthat is in place at the time the selecting finger set of fingers isdetermined for the gesture, which is normally when at least one of thegesturing fingers lifts. This approach forces the user to learn tosynchronize holds with a specific phase of the gesture and gives theuser the duration of the gesture to decide whether to apply a hold.However, in giving the user the freedom to accidentally touch thegesture surface, the user may be forced to slow the rate of input inorder to properly establish a hold.

Only shift holds further parameterize the gesture. A shift holdparameterizes the gesture by the type of hold it represents, which isgiven by the finger identities and touch counts of the hold fingers.Under the gesture parameterization model for characterizingfinger-mapped gestures, the gesture system assumes a default value forthe shift hold parameter when no shift hold is active. Each value of ashift hold parameter identifies a “shift mode” of the gesture system.This makes the user request a function of the shift mode, regardless ofwhether the user is actively performing a shift hold. Identical gesturesinput in different shift modes may indicate different user requests.

It is possible for the gesture system to allow different fingers on asingle input region to simultaneously contribute to both a gesture and ahold. This is especially valuable for allowing the thumb to share aregion with the index finger so that the thumb may perform holds whilealso maximizing the space available to the index finger for gesturing.As previously explained, the gesture system should resolve fingeridentity conflicts among fingers by the time they begin gesturing.Because the gesture system need not resolve such conflicts for a fingeruntil the finger begins gesturing, there need not be a conflict betweengesturing and non-gesturing fingers on the same input region. Thegesture system is free to assign identities to fingers that have notbegun gesturing, even assigning them arbitrarily if necessary, shouldmore than one non-gesturing finger be touching a given input region.Each of these non-gesturing fingers can contribute to a hold if theyhave not gestured by the time the selecting finger set is establishedfor the gesture.

The finger maps in FIGS. 7B, 7F, and 7J-7N illustrate support for holds.The double-headed arrows (e.g. arrows 87-89) indicate regions (e.g.75-77) supporting input by gesturing fingers, and the dots (e.g. dots90-97) indicate regions that support holds. Assuming that these fingermaps are intended for a user's right hand, the finger map in FIG. 7Fmight support the thumb, the index finger, and the middle finger, withthe thumb being able to perform a hold, such as a shift hold. The fingermaps of FIGS. 7J-7N support thumb holds alongside a four-fingered map.The finger map in FIG. 7L supports either an index finger hold or athumb hold on the same region. FIGS. 7M and 7N show finger maps that aredesigned for use by both hands, with the hold region (containing dot 96or 97) supporting both thumbs.

FIGS. 33A-33H depict a number of hold gestures. Each figure depicts afinger map with two input regions 540 and 541. A series of dotsconnected by arrows (e.g. points 542-544) indicates a stroke gestureoriginating in one of the regions. A dot by itself (e.g. point 545 or546) indicates a finger performing a hold. A dot with a circle around it(e.g. point 546) is a finger with a touch count of 2 performing a hold.The resulting user request is shown to the right of each figure. Each ofthe depicted gestures except for that in FIG. 33G inputs a user requestfor inserting a character, which is shown in single quotes after theequals sign. The parenthesized word indicates the shift mode in whichthe session is operating. The figures show a series of gestures of agesture session. FIGS. 33A and 33B depict normal operation withoutholds. FIGS. 33C and 33D depict an anchor hold in effect on region 540;a single-touch hold on region 540 has no effect on the user request.FIGS. 33E and 33F show the effect of a single-touch hold on region 541,which causes the gestures to select digits instead of letters. In FIG.33G, an escape hold with touch-count 2 on region 540 temporarilydisables finger-mapped gesturing. The gesture shown in region 541 ofFIG. 33G is interpreted as something other than a finger-mapped gesture,with the touch count of the hold selecting the particular gesture systemwith which to interpret this gesture. When the hold finally lifts, thefinger map resumes with finger-mapped gesturing. Finally, FIG. 33H showsthat a hold with touch-count 2 on region 541 has a different effect fromthat on region 540: it causes the gestures to select user requests forinserting punctuation.

FIG. 31 refines the process for detecting a finger-mapped gesture withsteps for handling a shift hold. In addition to determining theselecting finger set in step 121, in step 530 the gesture systemdetermines whether a non-gesturing finger is touching and whether thefinger represents a hold. For example, the gesture system coulddetermine the finger identity of the non-gesturing finger and onlyrecognize certain finger identities as capable of indicating holds.According to test 532, if a hold finger is detected, the gesture systemdetermines a user request as a function of at least one property of thehold finger, in addition to determining the user request as a functionof the gesture parameters, as shown in step 531. If the gesture systemsupports different holds for different finger identities, the userrequest would be a function of the hold finger's finger identity. Forexample, the gesture system might support a shift hold for the indexfinger and a separate shift hold for the pinky finger. Alternatively,the pinky finger might provide a shift hold, while the index fingerprovides an anchor hold; in this case, the user request will be afunction of the identity of the finger performing the hold even thoughthe anchor hold does not alter the user request that the gestureotherwise indicates. It may also be that the user request is not afunction of the hold finger's identity but is a function of the holdfinger's touch count. Of course, the user request could be a function ofboth the hold finger identity and touch count. If no hold finger isdetected, the gesture system determines the user request as usual instep 123. Equivalently, the presence or absence of a hold finger mayrepresent a shift mode, with the user request always being a function ofthis shift mode.

7.7.4.3 Residual Holds

There is a drawback to only establishing hold fingers at the time theuser completes the gesture. While a user is rapidly entering gestures,there is a chance that the user may accidentally lift hold fingersslightly too soon, lifting them before the first gesturing finger haslifted. The more rapidly the user enters gestures, the more likely thisis to happen. Even so, it is possible for the gesture system to properlyinfer that a hold was intended for a gesture and apply it anyway. Thegesture system may accomplish this via “residual holds.” Establishing ahold at the time a gesturing finger lifts provides the user withflexibility in the timing of hold initiation, and residual holds providethe user with flexibility in the timing of hold completion.

A “residual hold” is a temporary extension of a potential hold after oneor more of the potential hold fingers have lifted. A residual holdstarts when a potential hold finger lifts. The residual hold thereafterrepresents the potential hold that was in place prior to the finger liftthat started the residual hold. The residual hold remains in effect fora period of time afterwards, a period of time called the “residual holdperiod.” The potential hold itself persists until the end of theresidual hold period; for the duration of this period, the potentialhold is a residual hold. If the user completes the selection of afinger-mapped gesture while the potential hold (and now also residualhold) is in effect, the potential hold is interpreted as a hold,parameterizing the gesture if it happens to be a shift hold. Restrictingthe duration of a residual hold both allows a user to prevent apotential hold from applying to a subsequent gesture and allows a userto separate a potential hold from the completion of a gesture by enoughtime to keep the potential hold from applying to the gesture. For thesereasons, it may be useful to allow the residual hold period to beconfigured for each user.

If the user is typing rapidly enough, it is possible that the user willbegin and complete a new gesture before the residual hold has expiredfor the preceding gesture. This would mistakenly apply the residual holdto the subsequent gesture. To prevent this problem, the gesture systemshould only allow a residual hold to apply to the gesture that is inprogress at the time the residual hold goes into effect. This simplestway to accomplish this is to cancel the residual hold whenever a fingernewly touches the input area. This way, the fingers that are touching atthe time of residual hold starts are the only ones that may contributeto a gesture to which the residual hold applies.

FIGS. 34A-34H illustrate holds. Each of these figures depicts a fingermap of three regions 555-557. The series of figures represents a seriesof events. Locations 558-572 are numbered in the order in which fingersfirst touch the locations. A solid dot (e.g. location 560 in FIG. 34A)represents a location where a finger is touching. When two solid dotsappear in a figure, two fingers are simultaneously touching. An open dot(e.g. location 558 in FIG. 34A) represents a location that a fingerpreviously touched in its path. An arrow connecting dots (e.g. vector573) indicates a finger's direction of travel while touching. FIGS.34A-34D illustrate the occurrence of a residual hold. In FIG. 34A, theuser has gestured a finger from point 558 to point 560, and in region556 and the user has touched a finger at point 559 in region 557. Thefinger at point 559 is not exceeding the gesturing threshold. In FIG.34B, the finger at point 559 lifts, initiating a residual hold timeoutfor the finger having a duration equal to the residual hold period.Meanwhile, the other finger continues to gesture in region 556. In FIG.34C, the finger in region 556 lifts before the end of the residual holdtimeout. The gesture system therefore treats the gesture in region 556as if it had been performed with a hold in place on region 557. If holdsin region 557 are interpreted as anchor holds, the gesture in region 556indicates a user request that is not dependent on the hold. However, ifholds in region 557 are interpreted as selecting a shift mode forinputting numeric characters, the gesture in region 556 indicates anumeric character. In FIG. 34D, another finger touches the input area atpoint 562, before the residual hold timeout has expired. However,because a residual hold only applies to a single completing gesture, theresidual hold is no longer in effect.

7.7.4.4 Locked Holds

It is an option for the gesture system to “lock” the hold. A “lockedhold” is a hold that persists for as long as some or all of the holdfingers remain touching the gesture surface. The fingers of a lockedhold cannot be interpreted as anything other than hold fingers until allof the fingers of the hold have lifted. The user keeps the fingers of alocked hold touching as he or she inputs additional finger-mappedgestures. If the locked hold is a shift, it parameterizes each of thesegestures as it did the initial gesture on which the locked hold wasestablished, using the same shift mode.

Locked holds prevent users from having to lift hold fingers after everygesture, but more importantly, they allow hold fingers to move withoutbreaking the hold. In particular, a finger of a locked hold remains ahold finger even if it exceeds a gesturing threshold; once a hold islocked, the user need not be concerned about keeping the hold fingerstationary while otherwise gesturing, freeing the user to gesture morerapidly with the remaining fingers.

The series of FIGS. 34E-34H illustrates the user performing a lockedhold. In FIG. 34E, the finger at point 564 is touching but not exceedingthe gesturing threshold. Meanwhile, another finger has moved from point563 to point 565. In FIG. 34F, the finger at point 565 lifts. Since thefinger at point 564 was touching without gesturing at the time thegesturing finger at point 565 lifted, the gesture system interpretspoint 564 as performing a hold and locks the hold. The padlock symbolnext to point 564 symbolizes the state of being locked. If the gesturesystem interprets holds in region 555 as anchors, the gesture thatlifted at point 565 is interpreted as if the finger at point 564 werenot touching. If the gesture system interprets holds in region 555 asselecting gestures that indicate punctuation characters, the gesturethat lifted at point 565 indicates a punctuation character. In FIG. 34G,the locked hold remains touching as another finger starts a gesture inregion 557. The locked hold finger has moved to point 569 and exceededthe gesturing threshold. In FIG. 34H, the locked hold finger has againmoved significantly, but it remains touching as the finger gesturing inregion 557 lifts. Even though the locked hold finger has begungesturing, the gesture system continues to treat it as a hold. If itwere an anchor hold, the gesture in region 557 would be interpretedindependently of the hold. If instead the locked hold indicatedpunctuation characters as before, the gesture in region 557 wouldindicate a punctuation character.

FIG. 32 depicts the process of receiving finger-mapped gestures refinedfor handling locked holds. In this depiction, the gesture systemmaintains a shift mode indicating the set of user requests that areavailable to a gesture performed in that mode. The absence of any holdfinger, or the presence only of anchor modes, indicates a default shiftmode. The presence of a hold finger selects a shift mode. The gesturesystem starts in a default shift mode and subsequently retains anindication of the active shift mode. Upon receiving a new finger-mappedgesture, as part of test 538, the gesture system ascertains whether ahold finger from a prior gesture remains touching. If a prior holdfinger is found to remain touching, the gesture system skips the step536 process of determining a new shift mode for the present gesture,leaving the previously determined shift mode unchanged. It is an optionfor the gesture system to fail to recognize the gesture if not allprevious hold fingers remain touching. If no prior hold finger remainstouching, the gesture system evaluates the shift mode according towhether the current gesture has input a new hold finger, as shown instep 536. Regardless, the gesture system determines the user requestaccording to the gesture parameters and the resulting shift mode, asshown in step 537. There are many possible representations of the shiftmode, including as a variable, as a record of active hold fingers, andas a mode-specific branch of the program logic. After issuing the userrequest in step 124 and waiting for the fingers of the selecting fingerset to lift in step 125, the gesture system resumes gesture detection instep 535, which excludes previously established hold fingers from theselecting finger set.

A residual hold cannot be locked because all hold fingers must betouching to enact a lock; keeping the fingers touching is how a userlocks a hold. However, the gesture system may extend a locked holdbeyond the lifting of a hold finger by allowing locked holds to survivetemporarily as residual holds.

7.7.5 Backbrushing

A “backbrush” is a gesture enhancement that further parameterizes alockstep gesture with a second indication of distance travelled: thedistance that a gesture travels after reversing its initial direction.In a backbrush, the user performs a lockstep gesture as usual, butinstead of lifting the finger or fingers at the end of the gesture, theuser reverses the direction of the gesture for some distance and thenlifts. The length of this reversal translates into “backbrush level,”indicating a range of lengths analogous to the determination of agesture level. The gesture outcome is further a function of thisbackbrush level. When a user applies a backbrush to a gesture, the useris said to be “backbrushing,” and a gesture so modified is said to be“backbrushed.” Backbrushing is especially useful for selecting analternate form of a character, such as for selecting the capital oraccented form of a character that the gesture would normally selectabsent the finger action for backbrushing. For example, a gesturewithout backbrushing might indicate the lower case letter ‘a’, while thesame gesture with a backbrush appended might indicate the upper caseletter ‘A’.

7.7.5.1 Determining Reversal Points

To determine the backbrush level, the gesture system first identifiesthe reversal points on the gestured paths. A “reversal point” is a pointon the path of a finger at which the finger reverses directions. Priorto detecting backbrushing, the gesture system identifies a reversalpoint on a finger's path by keeping track of the point it determines tobe the farthest point from the measurement origin. When the distancefrom this point to the finger's most-recent point differs by at least aminimum distance for some distance measure, the farthest point isdeclared the reversal point. This minimum distance is called the“minimum backbrush length.” Alternatively, and not necessarilyequivalently, depending on the distance measure, when the distancemeasure of the finger's most-recent point differs from the distancemeasure of the farthest point by at least the minimum backbrush length,the farthest point is declared the reversal point. Once the reversalpoint has been declared, the gesture is considered to be in reversal andthe reversal point cannot be subsequently reassigned. If the gesturesystem never detects a reversal for a finger, the gesture system canconsider the reversal point to be the point at which the finger lifts.

The gesture system may prevent the user from reversing a finger againafter backbrushing by monitoring the finger for a subsequent reversal.To monitor for a subsequent reversal, the gesture system treats thereversal point as a virtual measurement origin and repeats the processfor identifying a reversal point relative to this origin, as if noreversal point had yet been determined. However, if the gesture systemsubsequently detects what would be declared a reversal point, thegesture system has actually detected a second reversal, and the gesturesystem instead treats the gesture as unrecognized until the gesturecompletes. Monitoring for subsequent reversals is useful for forcing theuser to properly understand backbrushing and for allowing the host todisambiguate backbrush gestures from other sorts of gestures that itmight support that do allow multiple reversals. As an alternative tomonitoring just the distance between a potential reversal point and thecurrent point, the gesture system can determine backbrush levels fromthe lengths of extent vectors.

The approaches just described for identifying the reversal point are notcompatible with retraction. Superficially, retraction and backbrushingseem incompatible because they interpret reversals of directiondifferently. However, there are ways to allow a gesture system tosupport both at once. The simplest approach requires that the gesturesystem start a timer with the first finger that begins gesturing in agesture. If the user completes the entire gesture before this timertimes out, the gesture system interprets backbrushing as just described.If the gesture completes after the timer times out, the gesture is notinterpreted with backbrushing, allowing reversals to be interpreted asretraction. Under this approach, a user performs retraction by ensuringthat the gesture has a minimum duration, so that only fast gestures canbackbrush. Alternatively, the gesture system can initiate the timer fora finger when it is determined that a finger has begun backbrushing andrequire that, once the finger has begun backbrushing, the gesture mustcomplete within a certain amount of time. Should the finger not completewithin this amount of time, the finger is interpreted as performing aretraction rather than a backbrush. As a final alternative, the gesturesystem can require that a backbrush achieve the minimal backbrush lengthat a certain speed or greater, such as calculated by the time differencebetween the point of the greatest distance measure and the point ofreaching the minimal backbrush length. Notice that none of theseapproaches is conducive to supporting retraction for the backbrushingportion of a gesture.

7.7.5.2 Initial and Reversal Lengths

The backbrush level is based on a length that the gesture is determinedto have moved in a reverse direction. If the gesture also includes agesture level parameter, the gesture system must determine two lengthsfor the gesture—an initial length and a reversal length—with the gesturelevel being determined from the initial length. As with determining thegesture length for the gesture level parameter, several approaches areavailable for determine the reversal length.

Under a straightforward approach, an initial distance is assigned toeach gesturing finger as the distance measure between the finger'smeasurement origin and its reversal point, and a reversal distance isassigned as the distance measure between the reversal point and thepoint at which the finger lifts. The initial length of the gesture isthen a function of the initial distances of the gesturing fingers, andthe reversal length of the gesture is then a function of the reversaldistances. Additional methods for determining the reversal length areavailable when a reversal point is treated as a measurement origin,because the methods explained previously for determining gesture lengthbecome available for determining reversal length. Under these methods,the finger lift points serve as the level selection points.Alternatively, the gesture system may monitor the extent vectors of thegesturing fingers. The initial and reversal lengths can be defined interms of the lengths of the first and second extent vectors,respectively. It is an option for the gesture system to decline torecognize a gesture as a backbrush upon detecting a third extent vector.

7.7.5.3 Determining Backbrush Level

The gesture system maps different ranges of the reversal lengths todifferent backbrush levels. When a gesture also includes a gesture levelparameter, the gesture system may opt to employ one function fortranslating initial lengths to gesture levels and another function fortranslating reversal lengths to backbrush levels. In order to simplifydiscussion, this specification identifies backbrush levels by integers.However, as with gesture levels, the gesture system need not identifybackbrush levels by integer. It can instead identify backbrush levels byany form of enumeration or even by associating different program logicwith different backbrush levels without maintaining a variable for thebackbrush level. If none of the fingers of the gesture backbrush, thegesture system can either treat the gesture as one not parameterized bya backbrush level or it can treat it as one parameterized by a “defaultbackbrush level” value. This specification uses 0 as the defaultbackbrush level, 1 as the first backbrush level, and 2 as the secondbackbrush level. To be clear that the accompanying claims are notrestricted to the representation of backbrush levels as integers, theclaims may refer to the lowest (non-default) backbrush level lengthrange as the “first-range” backbrush level and the second lowest lengthrange as the “second-range” backbrush level. It is up to the specificgesture system how to handle gestures for which some fingers backbrushand some do not.

The methods for determining a backbrush level are analogous to those fordetermining a gesture level. The gesture system can either determine asingle reversal length for the entire gesture, or it can determinemultiple intermediate reversal lengths for the gesture, such as one foreach finger. In the former case, the backbrush level is determined fromthe single gesture length, and in the latter case, an intermediatebackbrush level is determined for each intermediate reversal length anda backbrush level for the entire gesture is determined from theseintermediate backbrush levels.

FIG. 23 was described previously for gesture levels, but it alsoillustrates backbrush levels. The path tree for each finger A and Bdepicts three possible paths. The first path possibility (path 402 or405) consists of only a solid leg, ending at either point 375 or 384.This is the “solid path” of a path tree. The second path consists of asolid leg followed by a dashed leg (path 403 or 406), ending at point378 or 386. This is the “dashed path.” The third path consists of asolid leg, followed by a portion of a dashed leg, followed by a dottedleg (path 404 or 407), ending at point 380 or 388. This is the “dottedpath” of a path tree. Points 375 and 384 represent potential reversalpoints on the dashed paths 403 and 406 and on the dotted paths 404 and407. Arcs 392 and 393 indicate distances (D) and (E) from point 375,respectively, and arcs 397 and 398 indicated distances (I) and (J) frompoint 384, respectively. The columns in the tables of FIGS. 24A-24Cindicate distances from these potential reversal points.

Consider the following scenarios according to FIGS. 23 and 24A-24C:

-   -   1) Finger A gestures alone and lifts at point 378. Upon reaching        point 376 during this gesture, the gesture detects that the        finger has reversed directions and that point 375 is the        farthest point so far reported. Distance (D) corresponds to the        minimum backbrush length, so when the finger lifts at point 378,        it clearly has exceeded that distance from point 375 while still        not being the farthest point from the measurement origin 370.        The gesture therefore indicates a backbrush. Since backbrushes        are not detected at distances less than (D) from the reversal        point 375, distance (D) is the lowest possible backbrush        distance and hence the minimum distance for backbrush level 1.        However, the reversal to point 378 exceeds distance (E),        indicating a backbrush level of 2. Since point 375 was the        reversal point, the gesture level is determined for that point,        giving it a gesture level of 3 and a backbrush level of 2. The        table in FIG. 24A therefore indicates that this gesture inputs        the character    -   2) Finger A gestures alone and lifts at point 380. Upon reaching        point 376 during this gesture, the gesture detects that the        finger has reversed directions and that point 375 is the        farthest point so far reported. Distance (D) corresponds to the        minimum backbrush length, so the gesture has not backbrushed at        point 376. Point 379 is the next reported point, and it is        farther from the measurement origin 370 than point 375, so point        375 ceases to be a potential reversal point and there is no        potential backbrush in progress. When the finger lifts at point        380, there was no backbrush, which is equivalent to a backbrush        level of 0. The table in FIG. 24A therefore indicates that this        gesture inputs the character ‘G’.    -   3) Finger B gestures alone and lifts at point 386. Upon reaching        point 385, the gesture system determines that point 384 is now        farthest from the measurement origin 381 and so the finger is in        reversal. However, by point 385 the finger has also traveled a        distance from 384 greater than distance (I), indicating that the        gesture is a backbrush and has a gesture level of 2. When the        finger lifts at point 386, it lifts between distances (I)        and (J) from point 384, giving it a backbrush level of 1. The        table in FIG. 24B indicates that this gesture inputs the        character ‘N’.    -   4) Finger B gestures alone and lifts at point 388. As before,        upon reaching point 385, the gesture system determines that the        gesture is a backbrush. By point 388, the finger has reversed        directions again by more than distance (K) from point 385, as        shown by having passed arc 399. However, the preceding backbrush        path had already exceeded the minimum backbrush distance (I) by        point 385. The gesture system may optionally detect the path to        point 388 as a second reversal. If the gesture system does not        define gestures involving additional reversals, it may opt to        fail to recognize this gesture.    -   5) Fingers A and B together perform a sweep gesture, lifting at        points 378 and 386 respectively. The fingers have already been        individually shown to be backbrushing upon reaching these        points. The gesture levels for fingers A and B were shown to be        3 and 2, respectively, and the backbrush levels were shown to be        2 and 1, respectively. If the gesture system uses the greatest        gesture level of the fingers in a sweep, the gesture level is 3,        and if it uses the greatest backbrush level, the backbrush level        is 2. The table in FIG. 24C therefore indicates that this        gesture inputs the character ‘&’.

The backbrush gestures in FIG. 23 are examples of multi-levelbackbrushing. It is also possible for a gesture system to support onlyone backbrush level, so that each gesture is parameterized as eitherbackbrushing or not, without regard to backbrush level. FIG. 23 can beinterpreted as representing single-level backbrushing by equatingbackbrush levels 1 and 2, making the columns for backbrush levels 1 and2 identical in each of the tables 24A, 24B, and 24C. This is equivalentto dropping the backbrush distances (E) and (J), leaving only thecolumns “<(D)”, “<(I)”, and “<(D/I)” in the tables of FIGS. 24A, 24B,and 24C, respectively.

7.7.6 Multi-Reversal

The backbrushing gesture can be generalized into a gesture parameterhaving an arbitrary number of reversals. These gestures are“multi-reversal gestures.” A multi-reversal gesture furtherparameterizes the finger-mapped gesture by the number of reversalsperformed. Each reversal of a finger can be detected as a backbrush forthe current leg of the finger's path, in which case the number ofreversals is the number of backbrushes. The reversals can also bedetected by counting extent vectors, in which case the number ofreversals is the number of extent vectors minus one. The number ofreversals is the “reversal level.” If there is only one finger in theselecting finger set, the reversal level of the gesture is that of thefinger. If there are multiple fingers in the selecting finger set, thereversal level of the gesture can be determined from the reversal levelsof the individual fingers in one of the ways that the touch count of agesture can be determined from the touch counts of the individualfingers.

A multi-reversal gesture can indicate both a gesture level and abackbrush level. The gesture level can be determined as usual for abackbrushing gesture, using measurement up until the first reversal. Thebackbrush level of a finger can be a function of the lengths of thesubsequent legs of the path of the finger, such as their average, theirmaximum, or their minimum. This allows a user to perform a stroke or asweep and finalize it with a tickling-like gesture of varying butintentional average length, where the average length further indicatesthe user request selected. Alternatively, and perhaps more intuitivelyfor the user, the gesture level can be assigned to be a function of thelengths of all of the legs of the finger paths, so that the user is nothaving to distinguish the first leg from the following legs. In thiscase, the gesture has a reversal level but no backbrush level. It isalso an option to determine the gesture level as a function of all butthe last leg of each finger's path and then to determine the backbrushlevel from just the last legs, as usual for a backbrush gesture.

Simpler interpretations of multi-reversal gestures may be morepractical. Under the simplest interpretation, all multi-reversalgestures for a particular set of finger identities yield identicalgesture outcomes. This allows the user to think of selecting a userrequest simply by scratching the finger back and forth, without worryingabout the directions or the lengths of the segments of finger movement.A slightly more capable implementation ignores gesture direction andgesture level and instead selects a user request as a function of thedirection of movement of the final segment of the multi-reversal. Thisallows the user to think of selecting a user request by scratching backand forth and then ending the gesture in a particular direction toindicate the choice. This approach keeps the user from having to worryabout the direction in which the gesture starts or the number ofreversals performed. Under another alternative, a completion level canbe assigned to the gesture according to the length of the finalreversing segment of the gesture, and the user request can further be afunction of this completion level. The latter approach is useful becauseit may be easier for a user to scratch the finger a number of timesbefore consciously indicating a level for the gesture, rather than toindicate a level at the start of the gesture.

7.8 Mode Gestures

A “mode gesture” is a finger-mapped gesture that the user performs tochange the mode of the gesture system for subsequent finger-mappedgestures. Different finger-mapped gestures can be assigned to selectdifferent modes. The user request that any particular finger-mappedgesture indicates is a function of the current mode of the gesturesystem. Whereas a shift hold is analogous to holding the shift key downon a keyboard while pressing another key, a mode gesture is analogous topressing the “caps lock” key on a keyboard to change the charactersindicated by subsequent presses of letter keys. One mode gesture mightput the gesture system into a particular mode, while another mightreturn it to the previous mode. While some mode gestures might select aspecific mode, it's also possible to have a mode gesture return thegesture system to the previous mode regardless of what the previous modewas.

A mode gesture can also put the gesture system into a temporary modethat automatically ends. Analogous to features offered by some computeroperating systems, a first gesture can select an accent that is to beapplied to the next gesture. For example, if the first gesture selectsan umlaut and the next selects the letter ‘u’, the indicated characteris ‘ü’, and subsequent gestures proceed in the mode that preceded theumlaut gesture. It's also possible to require that multiple subsequentgestures be input to select the intended user request.

Other kinds of temporary modes are possible. The touch count of thegesture that selects the mode can indicate the number of followinggestures that are to be interpreted within the mode. In this case, thegesture selects the mode independently of the touch count, and eachsubsequent gesture within the mode represents a distinct user request.This approach allows the user to select a mode for multiple gestureswithout having to finish by entering another gesture to end the mode. Amode can also be made temporary, ending without having to input aspecialized mode-change gesture, by supporting single gestures thatsimultaneously indicate a user request and change the current mode. Inthis case, the gesture that ends the mode can institute a new mode orjust return the gesture system to the mode within which the current modewas initiated.

7.9 Generic Hand Calibrations

The means so far described for determining finger identities from inputregions can be generalized to encompass a variety of similar approaches.Consider that when determining a channel-based finger map, everylocation on the input area belongs to the region associated with thechannel line nearest to the location. A channel line is a type ofgeometric element. The principle works for arbitrary geometric elements:every location on the input area can belong to the region associatedwith the geometric element nearest to the location. Each geometricelement has an input region, but because input regions are only used todetermine finger identities, corresponding geometric elements withfinger identities makes regions unnecessary. It's sufficient todetermine the identity of a finger by identifying the geometric elementnearest to a location for the finger. The method used to identify thenearest geometric element can also vary.

7.9.1 Alternative Geometric Elements

Hand calibrations can be based on a variety of geometric elements otherthan channel lines. Consider the examples shown in FIGS. 43A-43D, eachof which depicts three regions 695-706 comprising an input area. Inthese figures, solid dots (e.g. dots 719-724) each represent a locationof a geometric element, and open dots (e.g. dots 725 and 734) eachrepresent an arbitrary location being tested to identify the geometricelement nearest to it. Solid dots that are circled (e.g. dots 719-721)represent locations on a geometric element designated as “forwardlocations.” Solid dots that are not circled (e.g. dots 722-724)represent designated “backward locations” of a geometric element. Acircled asterisk (e.g. asterisks 732 and 737) represents the averagelocation of the forward locations of a geometric element, and anasterisk not in a circle (e.g. asterisks 733 and 738) represents theaverage location of the backward locations of a geometric element.

FIGS. 43A-43D depict the following geometric elements:

-   -   Two-Location Geometric Elements—FIG. 43A depicts three regions        695-697 each defined by a geometric element consisting of        exactly two locations. The two locations of a geometric element        represent the average forward location (e.g. location 719) to        which a finger travels while gesturing and the average backward        location (e.g. location 722) to which a finger travels while        gesturing. The input area is partitioned so that each location        on the input area belongs to the region that corresponds to the        nearest geometric element. That is, for every tested location on        the input area, the gesture system determines the location of a        geometric element that is nearest to the test location and        assigns the test location to that geometric element. Consider        the dashed line segments 726-731 originating at test location        725. Line segment 726 is the shortest, which places the test        location 725 in region 695.    -   Multi-Location Geometric Elements—FIG. 43B depicts a        generalization of FIG. 43A, with each geometric element        consisting of an arbitrary number of locations. The input region        of each of these geometric elements is the set of locations that        is closer to a point in the geometric element than to any point        in any other geometric element. The figure was drawn by visually        estimating the boundaries between adjacent regions 698-700,        rather than by computing boundaries precisely with a computer.        It appears that having more points clustered together within a        geometric element helps to smooth the boundaries between        regions. However, it is computationally expensive to test        locations (e.g. location 734) on the input area against so many        geometric element locations, so the gesture system is better off        first approximating these geometric elements with simpler        geometric elements and then testing locations on the input area        against the simpler geometric elements.    -   Line-Segment-Collection Geometric Elements—FIG. 43C depicts        geometric elements that each consist of a collection of line        segments. For example, the geometric element of region 701        consists of line segments 735 and 736. The nearest geometric        element to a location on the input area is defined as the        geometric element whose bisecting line 713-715 is nearest to the        location. The bisecting line of a geometric element is a line        (e.g. line 713) that passes through both the average location of        the geometric element's forward locations (e.g. location 737)        and the average location of the geometric element's backward        locations (e.g. location 738). In other words, the gesture        system first translates these geometric elements into channel        lines 713-715 and then performs the nearness test as is standard        for a channel-based finger map.    -   Polygon Geometric Elements—FIG. 43D depicts geometric elements        745-747 that are polygons. The region boundaries 743-744 in this        figure are defined so that they bisect the space between the        nearest sides of adjacent polygons. For example, boundary 743        bisects the space between polygon sides 739 and 740, and        boundary 744 bisects the space between polygon sides 741 and        742. Under this approach, it is not sufficient to test a        location for the nearest polygon. Instead, the nearness test can        be performed by comparing locations to the input regions        associated with the geometric elements. In this figure, each        location on the input area is nearest to the geometric element        745-747 whose input region 704-706 contains the location. In        other words, testing a location for the region that contains it        is a form of testing a location for the geometric element        nearest to it. Alternatively, the nearness test can be construed        as testing for the nearest line that is coincident with a side        of a geometric element, with the test indicating the geometric        element having this side.

7.9.2 Determining Finger Identities

Each geometric element of a hand calibration corresponds to a fingeridentity, just as each channel line of a channel-based hand calibrationhas an associated finger identity. A generic means for assigning afinger identity to a finger at a location is to identify the geometricelement that is nearest to the location. Because geometric elements areonly constructs for determining finger identity as a function oflocation, they need only be virtually located on the gesture surface andneed not be completely or even partially contained within the inputarea. However, each geometric element “applies” to a facet of the touchdevice, that facet being the one for which it is used to associatelocations on the facet with finger identities. Every geometric elementhas an associated region on the facet to which it applies. All locationsin that region are considered to be nearer to that geometric elementthan to all other geometric elements applying to the same facet.

The geometric form of a geometric element's region is called the “regionform” of the geometric element. A geometric element may be identical toits region form, allowing a region to itself be considered a geometricelement. In this case, a hand calibration may dynamically determine theform of the geometric element, although doing so likely entails basingthe new form on fixed underlying geometric elements. For example,channel lines can either serve as geometric elements themselves or theycan be used to calculate regions that are then used as the geometricelements for nearness determinations. As another example, consider thefinger maps in FIGS. 7D-7L. The boundary line to the left of each regionin these finger maps (e.g. any of boundary lines 82-86) could serve as ageometric element, with the nearest geometric element to a locationgiven as the nearest geometric element to the left of that location. Thegesture system could perform nearness tests with these left boundarylines, or it could use these boundary lines as the underlying geometricelements of dynamically sized quadrilateral geometric elements withwhich it performs nearness tests.

Nearness tests can be generalized to identifying the nearest geometricelement to a path of a finger, basing the tests more generally on one ormore points in the paths of fingers. Basing nearness exclusively on afinger's initial touch location—or a finger's originating region—isproblematic when two fingers each have an initial touch location nearestto the same geometric element, since both fingers should not be assignedthe same finger identity. More sophisticated nearness tests such as thefollowing are also possible:

-   -   1) The nearness test determines the geometric element nearest to        the location on the finger's path at which a “finger        identification threshold” is reached. The finger identification        threshold can occur upon exceeding a minimum distance from the        initial touch location or upon exceeding a minimum amount of        time from the initial touch point. FIG. 45A illustrates this        option on a finger map with three regions 777-779. The figure        depicts the path 774 of a finger whose initial touch location        771 is in region 777. The identification threshold 773 occurs at        point 772 in region 778, therefore assigning the finger to the        finger identity to which region 778 or its associated geometric        element corresponds.    -   2) The nearness test determines the geometric element nearest to        the path of a finger, computing nearness as a function of the        distances of multiple points on the path to adjacent geometric        elements. FIG. 45B illustrates this option in combination with a        finger identification threshold on a finger map with three        geometric elements 780-782. The dotted lines (e.g. lines        775-776) represent distances from locations on the finger's path        774 to the adjacent geometric elements 780 and 781. In this        example, only the points up to and including the point 772 at        the identification threshold 773 are used in the computation.        The sum of the distances from the points to geometric element        780 are compared to the sum of the distances from the points to        geometric element 781. In this case, the sum for geometric        element 781 is less than that for 780, indicating that the        finger's path is nearest to geometric element 781.    -   3) The nearness test determines the geometric element nearest to        the majority of points on a path, according to the count of        points, either for the entire path or for the portion of the        path up to an identification threshold.    -   4) The nearness test determines the geometric element nearest to        the average of the points on the finger's path or nearest to a        midpoint on the finger's path.    -   5) The nearness test determines the geometric element whose        region contains the longest length of the finger's path.

In addition to determining the nearest geometric element to a finger'spath, the gesture system can determine the second-nearest geometricelement to a finger's path. This is the geometric element to which thefinger's path is nearest assuming that the nearest geometric elementdoes not apply to the facet. The second-nearest geometric element can beused for conflict resolution when multiple fingers are nearest to thesame geometric element. It is possible that multiple geometric elementswill be equally nearest to a path or equally second-nearest to a path,in which case the gesture system can choose one of the geometricelements by arbitrary logic or else just decline to recognize thegesture. FIG. 19 illustrates a generic process for performing conflictresolution with arbitrary geometric elements. It determines thegeometric elements that are nearest to each finger of the selectingfinger set in step 270 and handles two-finger conflicts analogously tothe handling of conflicts for geometric elements that are identical totheir input regions, except that geometric elements are identified bynearness tests and not necessarily by the input region in which eachfinger has an initial touch location. Steps 271 and 272 handle the casein which the fingers are nearest the same geometric element, and step273 handles the case in which the fingers are nearest to differentgeometric elements. It is not necessary that a gesture system performconflict resolution.

The process for detecting finger-mapped gestures can be refined for handcalibrations that determine finger identities from geometric elements.FIG. 10 depicts this process. It is a refinement of FIG. 9. In step 130,the user first provides a hand calibration specifying properties ofgeometric elements. In step 131, a finger map is prepared from thegeometric elements, with this finger map forming the input area. Afterdetecting the selecting finger set in step 121, the gesture systemdetermines the selected identity set in step 132 from the selectingfinger set based at least in part on the nearness of fingers of theselecting finger set to the geometric elements. It is possible for thehost to provide some of the finger identities, so the entire selectingfinger set need not be determined according to the nearness of fingersto geometric elements. The remainder of the process, steps 123-125,proceeds as in the general case of FIG. 9.

When the host provides all of the finger identities, geometric elementsare not needed to identify the fingers. In such a gesture system, afinger of any finger identity may begin gesturing anywhere on the entireinput area. However, the input area is itself a geometric element. Asolution that determines finger identities by nearest geometric elementand a solution that receives finger identities from the host thereforehave the following property in common: a geometric element can beassociated with each finger identity such that a finger of that fingeridentity may begin gesturing at any location of the geometric element.When the host provides finger identities, this geometric element is theinput area and all finger identities are associated with the samegeometric element. When finger identities are determined by nearestgeometric element, this element is the input region of the geometricelement. In each case, this common property identifies an “isogeometricelement.” An “isogeometric element” of a finger identity is a geometricelement in which a finger may begin gesturing at any location in thegeometric element and indicate that finger identity. A geometric elementthat “specifies” a finger identity is an isogeometric element of thefinger identity. The input region of a geometric element correspondingto a finger identity therefore specifies the corresponding fingeridentity, and the input area specifies each host-provided fingeridentity.

7.9.3 Properties of Hand Calibrations

The properties of channel lines and input regions can be generalized togeometric elements. As with channel lines and input regions, geometricelements are able to share a facet, to be positioned on a facet relativeto other geometric elements, to form correspondence series, and onpermitting touch devices, to be oriented relative to a facet. Handcalibrations based on geometric elements in general can therefore employthese properties in finger maps. Geometric elements generalizefinger-mapped gesturing to work with a broad variety of handcalibrations. The generalization includes both channel-based handcalibrations and input regions not based on underlying geometricelements.

Any geometric element can be associated with a position on the gesturesurface, and geometric elements can be positioned relative to oneanother. The portion of a geometric element that marks its position isimmaterial. For example, the position of a geometric element might begiven by the location of its center or by the location of a particularcorner of the geometric element. A hand calibration for a finger mapspecifies the relative positions of geometric elements and perhaps alsotheir locations on the gesture surface; specific locations necessarilyconvey relative positions. This information allows a hand calibration tobe positionable, orientable, and sizable on the gesture surface, makinga hand calibration is a generically reusable characterization of a hand.

Geometric elements can be approximated as channel lines so that theinput regions can be computed in the simple fashion of channel-basedfinger maps. FIGS. 43A-43D illustrate computations of approximatechannel lines 707-718 for a variety of geometric elements. A channelline can be approximated for any given geometric element as a line thatpasses through the average location of the forward locations and theaverage location of the backward locations, requiring that geometricelements have designated forward and backward locations. For example,channel line 707 passes through the average forward location 719 and theaverage backward location 722 of input region 695, and channel line 713passes through the average forward location 737 and the average backwardlocation 738 of input region 701. A forward direction can also beassociated with each geometric element for purposes of orientingdirectional partitionings and possibly also for determining discretedirections. As usual, the forward direction can be given by a forwardangle. Being lines, each approximated channel line has two angles ofcharacterization, differing by π radians. The forward angle is the anglethat points roughly in the direction of the average forward location.FIGS. 43A-43D depict the forward angles as θ_(A), θ_(B), and θ_(C).

Forward directions that are used to determine channel lines or otherwiseshape input regions need not dictate the forward directions of gestures.Consider FIG. 38. The fingers tend to gesture along the channel lines663-666. Each finger gestures with a different forward direction,suggesting that the gesture system could determine the discretedirection in which each finger gestures separately for each finger byorienting a directional partitioning to the finger's forward direction.However, this extra work may not be necessary when the host dictates thetop and bottom of the gesture surface. The gesture system can properlyidentify forward and backward directions for fingers gesturing alongchannel lines 663-666 simply by monitoring whether a gesture ends closerto the top or the bottom of the gesture surface compared to where thegesture began. For example, the gesture system could determine thedirection of the gesture simply by examining the change in the values ofthe vertical y-coordinates of the fingers during the gesture. In thiscase, the gesture system may employ per-finger forward directions toestablish a finger map, but it uses a constant forward angle for thegesture as a whole, as illustrated with reference direction 462 andreference angle θ₁ in FIG. 27A, and a directional partitioning isoriented to the reference direction 462.

7.9.4 Variety of Possible Hand Calibrations

A hand calibration is most abstractly a characterization of a hand thatdistinguishes it from another hand by establishing at least onerelationship among finger identities for fingers of the hand. Therelative spacing of geometric elements for finger identities, theorientations of geometric elements, and the forward directions of fingeridentities are the most important characteristics. Orientations andforward directions are ideally associated, leaving two primarycharacteristics: spacing and direction. This abstract definition allowsfor a broad variety of possible hand calibrations. A hand calibrationcan be as simple as a number specifying the spacing between fingers, orit can be detailed like a collection of channel line segments and restlocations.

It is possible to define the relative spacing and directions of everytwo adjacent fingers of a hand, but this is unnecessary. The touchdevice may require that the user employ a hand in a way that reduces thenumber of characteristics required for a hand calibration. ConsiderFIGS. 4A-4D. The barrier 19-22 between two fingers in any of thesefigures forces the hand into a particular orientation and likelyeliminates the need to determine the relative positions and relativedirections of fingers that are immediately adjacent to the barrier. Inaddition, the characteristics of some fingers may reasonably be inferredfrom the characteristics of other fingers.

FIGS. 46A-46F depict a variety of ways to infer the spacing betweengeometric elements in a finger map. Each figure depicts a different kindof hand calibration for the same set of regions 806-809, with each handcalibration positioning the geometric elements and input regions of afinger map. Hard-coded boundaries between regions are given as solidlines 784, 785, 788, 792, and 793, while dashed lines (e.g. lines786-787 and 789-791) indicate derived boundaries between regions. Eachsolid dot 794-800 represents a specified position for a geometricelement. Each open dot (e.g. dots 801-804) represents an inferredposition for a geometric element. Each dotted line (e.g. line 805)indicates an orientation line along which the gesture system distributesgeometric elements. The letter U denotes a unit length inferred from theprovided geometric element positions. The figures depict handcalibrations as follows:

-   -   FIG. 46A. This finger map forces the user to place a finger in        the predefined region 806, whose size and position are fixed.        Only a single position 794 for a geometric element is specified.        The remaining geometric element positions 801-802 are evenly        distributed between the fixed region 806 and the provided        position 794.    -   FIG. 46B. Here the positions for the geometric elements 795-796        at the ends of an correspondence series are specified. The        positions 803-804 of the intervening geometric elements are        interpolated and evenly distributed.    -   FIG. 46C. This finger map depicts both interpolating and        extrapolating positions for geometric elements, given the        positions 797-798 of two provided geometric elements.    -   FIG. 46D. This finger map makes it clear that it's possible to        extrapolate multiple geometric elements distant from the ones        for which positions are specified. Positions 799 and 800 are        here specified for input regions 806 and 807, respectively. This        approach may introduce too much error to be effective.    -   FIG. 46E. This finger map demonstrates defining four input        regions 806-809 from a single characterizing value H, which        represents the hand width of the finger map in arbitrarily        selected units. The constant K converts those units to a        distance appropriate for the touch device. The input regions        806-809 must be positioned relative to some fixture. This finger        map is centered on the surface allocation, which is equivalent        to fixing boundary 792.    -   FIG. 46F. This finger map specifies the relative sizes of each        input region via the constants J, K, V, and W. The finger map is        otherwise analogous to the finger map of FIG. 46E, requiring        only that a single value H be specified. The finger map is        illustrated here with a fixed boundary 793 between regions 806        and 807.

FIGS. 46A-46F also suggest the variety of ways there are to describe theposition of a geometric element. Recall that a solid vertical linerepresents an unchanging boundary of a region. For example, in FIG. 46B,region 806 has a fixed left boundary 784 and region 809 has a fixedright boundary 788. If the positions of the regions were given bylocations for their left boundaries, the position of region 806 in thisfigure would never change, but the position of region 809 would change,because its left boundary is the varying boundary 791. However, ifregion positions were given by the right boundaries, the position ofregion 809 would never change because its right boundary is the fixedboundary 788, unlike the position of region 806, which would be given bythe varying right boundary 789. Consider also FIG. 46E, in which theboundary 792 between regions 807 and 808 is fixed. If the position of aregion were given by a location for its left or right boundary, only oneof these two regions would have a fixed position. In each of thesescenarios, at least one boundary of each region changes as handcalibrations vary, except for regions that are completely fixed,demonstrating that hand calibrations position regions relative to oneanother.

FIGS. 47A-47F depict a variety of ways to infer forward directions for aseries of finger identities, expressing forward directions as angles.Each figure depicts the forward angles of a hand calibration for fourfingers, with each figure depicting a different kind of handcalibration. The vectors 815-818 represent the forward directionsassociated with each finger, where the angle of a vector is the forwardangle of the forward direction. The mathematical expression next to eachforward angle indicates the value of the forward angle in radians. Boldvectors (e.g. vector 815 in FIG. 47A) represent predetermined, fixedangles. When a figure depicts a vector as a solid line (e.g. vectors 815and 818 in FIGS. 47A and 47B), the figure's hand calibration specifiesthe vector's associated forward angle. When a figure depicts a vector asa dashed line (e.g. vectors 816-817 in FIGS. 47A and 47B), the forwardangles associated with the vector are inferred. The angle μ denotes aunit arc inferred from provided angles. The dotted line 819 in eachfigure represents an orientation line, which can have a fixedorientation relative to the touch device or can be oriented to coincidewith positions provided for geometric elements, if the hand calibrationassociates geometric elements with finger identities. The bold vector820 represents a perpendicular to the orientation line 819 and serves asa fixed angle relative to which the forward angles are computed. Thefigures are as follows:

-   -   FIG. 47A. This gesture system forces the user to gesture a        finger along a channel line coincident with vector 815. The        forward angle of vector 818 is the only forward angle specified.        The forward angles for the intervening vectors 816-817 of the        series are inferred and spaced equally apart across the range of        angles between vectors 815 and 818.    -   FIG. 47B. Here forward angles are specified for the vectors 815        and 818 representing end-of-series finger identities. The        forward angles of the intervening vectors 816-817 are        interpolated and evenly distributed between the angles of        vectors 815 and 818.    -   FIG. 47C. This hand calibration depicts both interpolating and        extrapolating forward angles, given the two specified forward        angles of vectors 815 and 817.    -   FIG. 47D. This hand calibration makes it clear that it is        possible to extrapolate multiple forward angles that are not        between the forward angles specified. In this figure, the        forward angles for vectors 815 and 816 are given, while the        forward angles for vectors 817 and 818 are extrapolated. This        approach may introduce too much error to be effective.    -   FIG. 47E. This hand calibration demonstrates defining four        forward angles from a single characterizing value λ, which is        the length of the arc that the forward angles are to span, given        in arbitrarily selected units. The constant K converts these        units to an angle. The forward angles must be oriented relative        to an orientation line 819. In particular, the orientation is        done relative to a perpendicular 820 to the orientation line.    -   FIG. 47F. This hand calibration specifies the relative lengths        of arc of the forward angles via the constants J, K, and V. The        hand calibration is otherwise analogous to FIG. 47E, requiring        only that a single value λ be specified. The angles δ, ζ, and ξ        shown in the figure are intermediate angle calculations. The        forward angles in this figure are also oriented relative a        perpendicular 820 to the orientation line 819.

This specification has described numerous ways to establish a handcalibration. To “establish” a hand calibration is to create a new handcalibration according to the characteristics of a hand, to select frommultiple previously existing hand calibrations according to thecharacteristics of a hand, or to indicate a hand calibration by refininga previously existing hand calibration. Calibration gestures areparticularly convenient techniques for establishing hand calibrations.

7.10 Gesture Properties

Finger-mapped gestures are characterized by a few important properties.In order to maximize the accuracy with which users can rapidly gesture,gesture parameters should exhibit “position independence” to eliminatethe need for users to carefully target where fingers begin and endgesturing. “Form dependence” is a way to provide a level of positionindependence by ensuring that the gesture outcome actually depends onhow the fingers travel while gesturing in a way that is at least in partindependent of where gestures start or end. “Longitudinal selectability”ensures that a user can input a variety of gestures by extending andflexing the fingers while keeping the hand stationary on the gesturesurface. By not requiring the whole hand to move to input gestures, theuser need not constantly reposition the hand between gestures, allowingfor a rapid rate of gesturing. “Longitudinal bias” associatesstationary-hand gestures with the user requests that should be inputmost rapidly, thus ensuring that the gesture system takes advantage ofthe property of longitudinal selectability. Finally, finger-mappedgesturing can be made especially efficient using “gesture lengthaccommodation” to size the input area to accommodate rapid patterns ofgesturing.

7.10.1 Position Independence

In order to allow the user to gesture rapidly without having to look atthe fingers, the gesture system should maximize the amount offlexibility that users have for where fingers can start and end gestureson the gesture surface while still indicating the correct gesture.Ideally, finger-mapped gestures should be independent of their positionsexcept as needed to specify finger identities. This property is called“position independence.” Consider two gestures that are identical inform but start in different locations, thus also ending in differentlocations. The two gestures should yield the same user request even ifone gesture begins at the bottom of an input region and the other beginsat the top, although in this example they may have to travel off of theinput area to complete the gesture. Likewise, they should yield the sameuser request if only one ends outside the input area, if only onecrosses region boundaries, or if one ends in a different input region.Position independence gives the user maximum room to vary whereidentical gestures are input while still indicating the desired userrequest. To provide a gesture with position independence, the values ofthe gesture parameters other than the selected identity set shouldthemselves be determined in a position-independent way, except to theextent that they depend on the selected identity set.

Although position independence is ideal, it turns out that it is tooconstraining to require that finger-mapped gestures beposition-independent. Consider that a gesture system may superimposemultiple gestures on the same input area so that it can interpretgestures differently depending on where the gestures start and end. Forexample, a gesture system might make application functions available vialong unwieldy gestures that start in the input area, perhaps accordingto input regions, and end somewhere outside the input area. If thegesture parameters include an indication of the portion of the gesturesurface in which a gesture ends, gesture level can be determined as afunction of where the gesturing fingers end. This makes it possible toprovide two different determinations of gesture level depending onwhether the gesture represents an application function; users couldinput application functions at gesture levels that are much longer thanthose used for inputting text. It may be debatable whether suchapplication gestures are distinct from position-independent gesturesthat are also available. If these two forms of gestures were notconsidered to be separately parameterized, gesture level would be afunction of where the gesture ends and hence not independent ofposition. The potential for superimposing gestures complicatesassertions about position independence.

“Positional side gestures” are another example of gestures that agesture system may support in addition to position-independent gestures.In a “positional side gesture,” a region's longitudinal axis is dividedinto multiple subregions and gesturing left or right selects a userrequest as a function of the subregion in which the gesture begins. Inorder to also support efficient finger-mapped gesturing, such a gesturesystem should not distinguish the user requests of forward and backwardgestures as a function of subregion, as otherwise the user would have tofrequently look at the fingers to ensure that they were startinggestures in the correct subregion. Even so, the gesture system couldemploy a gesture parameter that identifies the originating subregion sothat gestures are not position-independent, while still making all userrequests assigned to forward and backward gestures identical for alloriginating regions. Once again, it is arguable whether positional sidegestures are separate gestures or simply parameterizations of forwardand backward gestures.

It is apparent that forward and backward gestures should at least be“origin-independent,” meaning that they should be independent of theirstarting locations. If they were not origin-independent, users would besomewhat randomly selecting user requests as the fingers stray duringsessions of longitudinal gesturing. Even so, forward and backwardsgestures can depend on their originating locations in ways that do notsignificantly affect the user. Consider a gesture system in which agesture level 2 threshold has a length of 100 pixels for gestures thatoriginate in the top half of an input area and a length of 101 pixelsfor gestures that originate in the bottom half of an input area. Usersare unlikely to notice a one-pixel difference in the behaviors of thetop and bottom halves of the input areas, and yet identical gestures oflength 100 pixels produce different results depending on where theyoriginate in the gesture area. Perhaps the most that can be said is thatforward and backward finger-mapped gestures are substantiallyorigin-independent.

The ideal of position independence does not suggest that all forward andbackward gestures that are identical in form and finger identity shouldyield a user request when any of the gestures can yield a user request.It only suggests that the user requests should be identical when they doyield user requests. Consider that either the operating system or thegesture system may preclude a gesture from yielding a value if itexceeds certain boundaries, such as the boundaries of the input area.Consider also that input regions are designed to accommodate the forwardand backward movement of fingers, so it is reasonable for a gesturesystem to fail to recognize a forward or backward gesture that venturesinto an adjacent region before completing. FIGS. 35A-35D and 36A-36Dillustrate position-independent gestures and are explained below.

Although it is unreasonable to require that gesture parameters beposition-independent, gesture level, gesture direction, and backbrushlevel are well-suited for implementation as position-independent gestureparameters, because these gesture parameters are defined in terms of theforms of the finger paths. However, to be useful, these gestureparameters must be implemented as substantially origin-independent.Touch counts and shift holds are position-independent because they areonly dependent on position through dependence on the selected identityset.

7.10.2 Path, Vector, and Form Dependence

There are many ways to parameterize a gesture, including input by meansother than the gesture itself. Finger-mapped gestures require that theuser move at least one finger on the gesture surface to select a valuefor at least one gesture parameter other than the finger identities. Atleast one of these gesture parameters must take advantage of the factthat the fingers have traced paths. Specifically, each finger-mappedgesture includes at least one “path-dependent” gesture parameter. Apath-dependent gesture parameter is a gesture parameter whose value is afunction of a path of at least one finger of a gesture's selectingfinger set. “Path dependence” just ensures that what the fingers dowhile gesturing helps determine the gesture outcome.

The property of “vector dependence” is stronger than path dependence. A“vector-dependent” gesture parameter is a gesture parameter whose valueis at least in part determined from the relative locations of distinctpoints in a path of at least one finger of a gesture's selecting fingerset. Recall that a “vector” represents the location of one pointrelative to another; the value of a vector-dependent gesture parameteris dependent on the properties of at least one vector of a path of thegesture. A vector-dependent gesture parameter is therefore at leastpartly dependent on the forms of the paths of the gesturing fingers,regardless of where the paths are on the gesture surface, thus impartingthe gesture parameter with a degree of position independence.

Gesture level, gesture direction, and backbrush level can be implementedas vector-dependent gesture parameters. These gestures parametersrequire that one or more fingers travel at least some distance, and thevalues of these gesture parameters depend on the relative locations ofpoints in the path or paths traveled. In the cases of gesture level andgesture direction, the relative locations of the first and last pointsof a path generally determine the value of the parameter. The value of abackbrush level parameter depends on whether a finger reverses directionduring the gesture, and reversals can be detected according to therelative positions of distinct points. The gesture system onlyimplements a gesture parameter in a vector-dependent way if it computesthe values of the gesture parameter from at least one vector in the pathof at least one finger.

Touch counts, shift holds, and shift buttons cannot be implemented asvector-dependent gesture parameters. The value of a touch countparameter depends on the number of times a finger touches and not on thepath that a finger travels while it touches. Whether or not a tap occursduring a touch count does depend on whether the finger follows a path,but the value of the touch count is not dependent on any single path. Ashift hold is not a vector-dependent gesture parameter because its valueonly depends on the actions of one or more fingers not in the selectingfinger set. Likewise, pressing a user interface button concurrently withperforming a gesture may influence the gesture outcome and thus serve asa gesture parameter, but the button is not vector-dependent.

The property of “form dependence” is even stronger than vectordependence. A “form-dependent” gesture parameter is a gesture parameterwhose value is determined at least in part from the form of at least onepath of the gesture's selecting finger set. All vector-dependent gestureparameters are form-dependent, but not all form-dependent parametersneed be vector-dependent. It may be possible to compute the value of agesture parameter through analogue means or via table look-ups notrequiring the computation of vectors. In a most abstract sense, gesturelevel, gesture direction, and backbrush level can all be implemented asform-dependent gesture parameters that are not vector-dependent byindexing all possible paths directly to the parameterizations of theassociated gestures. Whether a gesture parameter is form-dependent oronly vector-dependent, it is still possible to further determine thegesture parameter as a function of a position-dependent property of thegesture. However, the more form-dependent the gesture parameters are,the more efficiently the user is likely to be able to input thegestures.

FIGS. 35A-35D and 36A-36D illustrate gestures being performed on fingermaps. These gestures are both position-independent and vector-dependent.Each finger map has three input regions 575-577 identifying fingers A,B, and C. One or two finger paths are drawn on each finger map, eachfinger path designated by a series of dots connected by arrows (e.g. thepath from points 578 to 579, points 594 to 595, and points 596 to 597).The arrows indicate the finger's direction of travel. The first dot(e.g. points 578, 594, and 596) in an arrow-connected series representsthe finger's measurement origin. The last dot (e.g. points 579, 595, and597) in an arrow-connected series represents the location at which theuser lifted the finger from the gesture surface. The dashed lines (e.g.partitioning lines 585 and 600) partition the directions into aquarter-plane partitioning relative to a measurement origin (e.g. points578 and 605). Each circle (e.g. circles 583, 584, 601, and 602) centeredon a measurement origin depicts a minimum distance for a gesture level.The innermost circle (e.g. circles 583 and 601) represents a gesturingthreshold distance, so a finger must travel outside this circle toregister as gesturing in these examples. This is also necessarily theminimum distance for gesture level 1. Single quoted letters (e.g. ‘D’586 and ‘N’ 603) represent user requests. Each user request isassociated with a gesture level/discrete direction combination.

The finger maps of FIGS. 35A-35D depict strokes whose paths all haveidentical forms but start at different locations. The number of pointsin a path and their relative locations are the same in all of the fingerpaths. Each figure illustrates the process of determining a discretedirection and a gesture level for the gesture using a graphicalshorthand called a “target.” A “target” is a set of concentric circlesthat are centered on perpendicularly intersecting dashed lines. Forexample, the target 587 consists of the concentric circles 583 and 584,and they are centered on the intersecting dashed lines marked as 585.The concentric circles represent distances from the measurement originof the gesture. In FIG. 35A, the measurement origin 578 is at the centerof the concentric circles 583-584. The distance from the measurementorigin 578 to the inner circle 583 represents the gesturing thresholddistance. The space between the inner circle 583 and the outer circle584 is the range of distances associated with gesture level 1. The spaceoutside of the outer circle 584 is the range of distances associatedwith gesture level 2. The gesture of FIG. 35A starts at the measurementorigin 578 and ends at a location 579 in the gesture level 2 range ofdistances from the measurement origin 578. This gesture is therefore agesture level 2 gesture. This figure associates gesture level 2 witheach of the characters ‘B’, ‘F’, and ‘H’. However, the intersectingdashed lines 585 represent a quarter-plane directional partitioning ofthe directions that start at the measurement origin 578. These dashedlines 585 partition the space of possible directions into four discretedirections. For example, one discrete direction of target 587 includesthe characters ‘E’ and ‘F’. The gesture of FIG. 35A ends in the discretedirection that corresponds to the letters ‘A’ and ‘B’. The gesture levelindicated one of the letters ‘B’, ‘D’, ‘F’, and ‘H’, while the discretedirection indicated one of the letters ‘A’ and ‘B’. The only letter thatboth parameters together select is ‘B’, so according to target 587, thegesture of FIG. 35A is a request to input the letter ‘B’.

FIGS. 35A-35D use targets 587 and 588 to illustrate position-independentstroke gestures. In FIGS. 35A and 35B, the finger path begins in thesame input region 576, indicating finger B, but in two differentlocations 578 and 580. The directional partitioning 585 is the same, andthe gesture levels 583-584 are the same on each of the figures, mappingthe same discrete directions and gesture levels to the same userrequests. Both strokes thus indicate the user request ‘B’ according totarget 587. FIGS. 35C and 35D depict the same finger path in region 577,again starting at different locations 581 and 582. Although the form ofthe path is the same as that of FIGS. 35A and 35B, a different finger isperforming the gesture, finger C, so the discrete directions and gesturelevels of this gesture can map to different user requests, asillustrated in target 588. The finger paths begin at different locationsof region 577 in FIGS. 35C and 35D, but the gestures in these figuresstill indicate the same user request—the character ‘J’—because thegesture direction and the gesture level are here position-independentfor identical finger identities.

The finger maps of FIGS. 36A-36D depict two-fingered gestures for whichthe paths of different figures differ in starting location but haveidentical forms. The directional partitioning 600 and gesture leveldistances 601-602 together define targets 604 and 608, which differ onlyin the user requests that they indicate. FIGS. 36A-36D use targets 604and 608 to illustrate translating gesture parameters to user requests.The vector between the open circles (e.g vector 607 between open circles605 and 606) represents a vector interpolated between the finger pathsshown in the figure. This is an “interpolated distance vector,” which aspreviously explained, is one of several possible approaches to assigninga direction and length to a two-fingered finger-mapped gesture. Thefigures use this approach because it conveniently represents thedirection and gesture level of a two-fingered gesture visually as asingle vector. The forms of the finger paths are identical from figureto figure, yielding the same interpolated distance vector 607 indifferent locations, depending on the figure. It would have been equallyvalid to determine the gesture parameters by other means, such as byusing a gesture angle that is the average of angles of the fingers'distance vectors, and such as by using a gesture length that is theaverage of the fingers' distance vectors.

In each of finger maps of FIGS. 36A and 36B, one finger originates onregion 575 and the other on region 576, associating them with fingeridentities A and B and target 604. In FIG. 36A, the fingers start atpoints 590 and 591, and in FIG. 36B, the fingers start at points 592 and593, but because the finger paths are identical in form, the gestures inthese two figures both yield an interpolated distance vector 607 endingin the same parameterization of target 604. This parameterizationindicates user request ‘C’. When these same finger paths are positionedto originate in regions 576 and 577, as in FIGS. 36C and 36D, thefingers have different finger identities B and C. This potentiallychanges the user requests to which the gestures translate, because themapping of discrete directions and gesture levels to user requestsdepends on the combination of finger identities. Target 608 reflectsthis difference in user requests. FIG. 36C has the paths originating atpoints 594 and 596, and FIG. 36D has the paths originating at points 598and 599. According to these latter two figures and target 608, thesegestures both indicate a request to input the character ‘K’. Because thegestures depicted here are position-independent, geometricallytranslating the finger paths to the new locations does not change theresulting user request, provided that the fingers still originate in thesame regions.

Although the targets in FIGS. 35A-35D and 36A-36D uniformly distributecharacters among gesture levels and discrete directions, suchdistributions are not ideal for finger-mapped gestures. These figuresand other figures in this specification only depict targets that mapgesture parameterizations to user requests in order to simplify thegraphical representation and to assist with making the correlationbetween gesture parameterizations and user requests clear.

7.10.3 Longitudinal Selectability

“Longitudinal selectability” is perhaps the most important property offinger-mapped gestures for providing gesturing efficiency. Longitudinalselectability allows a hand to remain stationary while the fingers onlygesture longitudinally to select among user requests, allowing thefingers to input a rich variety of user requests despite only movingalong paths that approximate lines. Finger-mapped gestures providelongitudinal selectability by supporting “longitudinally selectable”gesture parameters. A “longitudinally selectable” gesture parameter is agesture parameter that distinguishes among multiple parameter values ina gesture for which one or more fingers travel along a set oflongitudinal paths that are available to the hand from a hand position.Longitudinal selectability can be defined geometrically in a way thatshould apply to the hands of a majority of users by defining it in termsof both “linear multi-selectability” and “longitudinal conformance.”

7.10.3.1 Linear Multi-Selectability

Longitudinal selectability requires that the user be able to inputmultiple parameter values for a gesture parameter despite gesturing thefingers only along paths that approximate lines. This property is called“linear multi-selectability.” Its definition is based on the lineardirections in which fingers can travel and on a set of“multi-selectability criteria” for gestures.

As suggested by the illustrations of longitudinal paths 615-618 in FIGS.37A-37B, longitudinal paths are lines or approximately lines, at leastover the spans of distances that are useful for efficient finger-mappedgesturing. Each longitudinal path can also be associated with a lineardirection representing the opposing directions of its approximatingline. In a linearly selectable gesture system, each finger that cangesture has at least one longitudinal path available to it. It ispossible to select a longitudinal path for each finger and to associatea linear direction approximation of the longitudinal path with thefinger's finger identity. An association of a linear direction with afinger identity is called a “linear association,” so a set of linearassociations may represent the forward and backward directions that areavailable to a set of finger identities. The gesture system need notrepresent linear associations as delineated entities. Linearassociations are inherent to gestures because the gesture systemassociates finger identities with gestured paths and because a gesturedpath can be characterized by one or more approximating slopes; at leastone linear association characterizes every possible gestured path.

One approximation of a linear direction of a path is the path's “simplelinear direction.” The “simple linear direction” of a path is the lineardirection represented by the path's “simple direction.” The “simpledirection” of a path is the direction of a vector that originates at theinitial touch location of the path and terminates at the path's finallocation. For example, in FIG. 40A, a finger gestures a path starting atlocation 635, passing through location 636, and ending at location 637.The vector 638 from the starting location 635 to the ending location 637defines the path's simple direction. The angle ϕ is thus arepresentation of the path's simple linear direction, as is the line639. The set of “simple linear associations” of a gesture is the set oflinear associations in which the simple linear direction of the gesturedpath of each finger of the gesture's selecting finger set is associatedwith a distinct finger identity of the gesture's selected identity set.A simple direction and a simple linear direction characterize a gesturedpath regardless of whether they are determined.

Longitudinal selectability depends on the availability of gestures thatmeet a set of “multi-selectability criteria.” Given an input area, a setof linear associations, and a gesture basis, the multi-selectabilitycriteria for a gesture are as follows:

-   -   1) The initial touch location of each finger of the gesture's        selecting finger is in the input area.    -   2) For each finger of the gesture's selecting finger set, the        simple linear direction of the finger's gestured path is a        linear direction of a distinct linear association of the set of        linear associations, and the gesture's selected identity set is        the set of finger identities of these distinct linear        associations.    -   3) The gesture basis characterizes the gesture. That is, the        gesture's selected identity set is identical to the        characteristic identity set of the gesture basis, and values are        determined for at least the gesture parameters of the gesture        basis.

A gesture parameter of a gesture basis is “multi-selectable” on a set oflinear associations within an input area if, and only if, the gesturesof the gesture basis that satisfy all of the multi-selectabilitycriteria together distinguish among multiple values for the gestureparameter. Recall that a gesture basis is a characterization of a classof similar gestures. It is possible to characterize the forward andbackward gestures of a gesture system with a set of gesture bases sothat, for a given input area and some set of linear associations, eachgesture parameter of each gesture basis is multi-selectable on the setof linear associations within the input area. It is therefore useful toextend multi-selectability to gesture bases and sets of gesture bases.Formally, a gesture basis is “multi-selectable” on a set of linearassociations within an input area if, and only if, each gestureparameter of the gesture basis is multi-selectable on the set of linearassociations within the input area. A set of gesture bases is“multi-selectable” on a set of linear associations within an input areaif, and only if, each gesture basis of the set of gesture bases ismulti-selectable on the set of linear associations within the inputarea. An assertion that a gesture parameter, a gesture basis, or a setof gesture bases is multi-selectable on a set of linear associations isin part an assertion that it is possible to gesture fingers so that thesimple linear directions of their paths are linear directions of the setof linear associations, but it is not an assertion that the gesturesystem must determine either linear directions for the gestured paths orlinear directions of the set of linear associations.

Because a simple linear direction can be established for any gesturedpath, the multi-selectability criteria do not require that fingersgesture along lines or approximate lines, and yet if a gesture parametercannot be input by gesturing along lines or approximate lines, thegesture parameter would not be useful for longitudinal gesturing. Due tothe diversity of possible device facet morphologies, and due to thevariety of possible gesture parameters, it is difficult to exactlycharacterize the class of gesture parameters for which multiple valuescan be selected while gesturing the fingers along lines orapproximations of lines. In particular, it is difficult to pragmaticallydefine what it means to be an approximation of a line. However, it ispossible to define specific classes of gesture parameters that makesense for lines or line approximations without requiring that a devicebe able to input gesture parameters of the class along lines orapproximations of lines. When a gesture parameter of one of theseclasses is multi-selectable on a set of linear associations, the gestureparameter is said to be “linearly multi-selectable” on the set of linearassociations and the gesture parameter is said to exhibit “linearmulti-selectability.”

This specification defines several classes of gesture parameters thatsupport linear multi-selectability. In particular, gesture level,gesture direction, backbrush level, and multi-reversal are generalclasses of gesture parameters that provide linear multi-selectability.Gesture direction provides linear multi-selectability when there is alinear direction for which opposite gesturing directions indicatedistinct discrete directions, while the remaining parameters inherentlysupport linear multi-selectability, provided that the gesture systemdoes not explicitly preclude interpreting these gesture parameters alonglines or paths that approximate lines. The two distinct discretedirections that are implicit with a multi-selectable gesture directionparameter of a gesture basis are each said to be “opposite” the other“in” the gesture basis. Although the gesture system is able todistinguish parameter values of a multi-selectable gesture parameter ona set of linear associations, it need not associate each possiblegesture parameterization with a user request, nor need it alwaysassociate distinct gesture parameterizations with distinct userrequests.

7.10.3.2 Longitudinal Conformance

The definition of “multi-selectability” does not constrain the lineardirections over which gesture parameters are multi-selectable. The mostefficient gestures are those in which the gesturing fingers travel thelongitudinal paths available to a hand in a hand position. A gesture inwhich the fingers move strictly along longitudinal paths is called a“longitudinal gesture,” and all other gestures are called “lateralgestures.” A gesture system should be characterizable by gesture basesthat are multi-selectable on a set of linear associations whose lineardirections approximate the forward and backward directions oflongitudinal gestures. The property of “longitudinal conformance”restricts linear directions to ranges of directions that seem reasonablefor the longitudinal gestures that most people's hands should be able toinput.

As fingers gesture longitudinally, they move along linear directionsthat are roughly perpendicular to the left-to-right orientation of thehand. For a hand in finger-mapping posture, the fingers are movingroughly perpendicular to a line that rest locations of the fingers wouldapproximate were the user to rest the fingers on the input area. Theorientation line of a hand calibration provides a good estimate of thisleft-to-right line. It is possible to define longitudinal gestures interms of the orientation line, but doing so suggests the presence of ahand calibration that provides rest locations for the end-of-seriesfingers, whereas longitudinal gestures are roughly perpendicular to theleft-to-right orientation of the hand regardless of whether thisinformation has been provided. Instead, it more generally applicable todefine the directions of longitudinal gestures in terms of theorientation lines that are potentially available to a hand.

Under this approach, the directions of longitudinal gestures areexpressed in terms of the rest locations that are potentially availableto the fingers at the ends of the series of fingers that can gesture,regardless of whether the gesture system has actually determined therest locations. The gesture system establishes a series of fingeridentities for the fingers of a hand, with the ends of the seriesrepresenting the fingers that are farthest apart. These are the fingeridentities that the gesture system associates with gesturing fingers.This information is sufficient to define longitudinal conformance. A setof linear associations is “longitudinally conformant” with a series offinger identities in an input area if, and only if, all of the followingcriteria are satisfied:

-   -   1) There are locations R₁ and R₂ in the input area and there are        end-of-series finger identities I₁ and I₂ of the established        series of finger identities, such that the gesture system is        able to detect a finger of identity I₁ touching at location R₁,        and such that the gesture system is able to detect a finger of        identity I₂ touching at location R₂.    -   2) Each linear direction of the set of linear associations can        be characterized by a direction angle that is within 25 degrees        of a direction angle for a line perpendicular to a line through        R₁ and R₂.

FIGS. 39A-39B illustrate this definition of “longitudinal conformance.”The input area of FIG. 39A is partitioned into regions 620-623, whilethe input area 624 of FIG. 39B is not. Each of these figures depicts aset of possible rest locations 625-628 and dashed lines 629-632 thatpass through the possible rest locations 625-628. Each dashed line629-632 represents a linear direction. Under one interpretation of thesefigures, the dashed lines 629-632 depict linear directions thatapproximate gestured paths. In this case, it is possible that a fingermay gesture along a dashed line, but it also possible that a finger ofthe appropriate finger identity may gesture along a different path, onethat has a simple linear direction identical to the linear direction ofthe dashed line. Under another interpretation of FIGS. 39A-39B, eachdashed line 629-632 is a channel line representing the expected oraverage forward and backward directions of gesturing fingers. Thefigures show a possible orientation line 633 passing through the restlocations 625 and 628 for the fingers associated with the end-of-seriesfinger identities. They also show a perpendicular line 634 to theorientation line 633, with the angle β₅ characterizing the perpendicular634. The angles θ_(A), θ_(B), θ_(C), and θ_(D) characterize the lineardirections of the dashed lines 629-632, and in all cases these lineardirections are within 25 degrees of the perpendicular 634. The lineardirections that the dashed lines 629-632 represent are therefore alllongitudinally conformant by the above definition, for this particularselection of rest locations 625-628.

The preceding definition is only a basic definition of “longitudinalconformance.” The definition helps restrict the range of lineardirections that are considered longitudinal, but it is not ideal,because a hand in a hand position gestures longitudinally with respectto a narrow range of possible orientation lines, whereas this definitionincludes the longitudinal linear directions of all possible orientationlines for all possible end-of-series rest locations. An observationhelps to refine this definition: were the user to position the hand atthe top or the bottom of the input area, the user would have to focusattention on the input area to be able to accurately start gestureswithin the input area. Effective use of the input area requires the userto place the fingers at a distance from the top and bottom. It istherefore reasonable to expect margins between the rest locations andthe boundaries of the input area. Establishing these margins reduces thespace of possible rest locations and thus further restricts the range ofdirections that are considered longitudinal.

Additional constraints can be added to the basic definition oflongitudinal conformance to assert margins. One constraint helps toensure that a finger can gesture the distances necessary to inputgesture level thresholds while minimizing the risk of starting gesturesoutside of the input area. A set of linear associations is“longitudinally conformant” with a series of finger identities in aninput area for Nth-level margins if, and only if, longitudinalconformance criteria (1) and (2) are satisfied and the followingcriteria are also satisfied:

-   -   3) There is a linear direction D₁ and a gesture level N        threshold length M₁ for finger identity I₁ such that R₁ is at        least a distance of M₁ from a boundary of the input area in each        direction of D₁ along a line through R₁.    -   4) There is a linear direction D₂ and a gesture level N        threshold length M₂ for finger identity I₂ such that R₂ is at        least a distance of M₂ from a boundary of the input area in each        direction of D₂ along a line through R₂.    -   5) Each of the linear directions D₁ and D₂ can be characterized        by a direction angle that is within 25 degrees of a direction        angle for a line perpendicular to a line through R₁ and R₂.

For example, when a set of linear associations is longitudinallyconformant with a series of finger identities in an input area forsecond-level margins, M₁ is the minimum length of the second-rangegesture level for finger identity I₁, and M₂ is the minimum length ofthe second-range gesture level for finger identity I₂. Likewise,third-level margins correspond to third-range gesture levels.

This definition of “longitudinal conformance” also has someshortcomings. First, the definition does not distinguish the top andbottom of the input area because “top” and “bottom” may be hard todelineate for some irregular or curved boundary input areas. Thisshortcoming is difficult to work around generically. Another shortcomingis that the definition is only applicable to gesture systems that usegesture level. The definition can be generalized by establishing marginsthat are independent of gesture level. A convenient way to do this is touse margins that are a percentage of the hand breadth. Fortunately, thedefinition of longitudinal conformance assumes rest locations and so canalso assume a hand breadth equal to the distance between those restlocations.

Under a generalization definition, a set of linear associations is“longitudinally conformant” with a series of finger identities in aninput area for N-percent margins if, and only if, longitudinalconformance criteria (1) and (2) are satisfied and the followingcriteria are also satisfied, where M is a length equal to N percent ofthe distance between locations R₁ and R₂:

-   -   6) There is a linear direction D₁ such that R₁ is at least a        distance of M from a boundary of the input area in each        direction of D₁ along a line through R₁.    -   7) There is a linear direction D₂ such that R₂ is at least a        distance of M from a boundary of the input area in each        direction of D₂ along a line through R₂.    -   8) Each of the linear directions D₁ and D₂ can be characterized        by a direction angle that is within 25 degrees of a direction        angle for a line perpendicular to a line through R₁ and R₂.

For example, when a set of linear associations is longitudinallyconformant with a series of finger identities in an input area for 15%margins, M is 15% of the length between locations R₁ and R₂. In theabsence of a hand calibration that specifies gesture level thresholds,it is reasonable to assume that a gesture level 2 threshold would be atleast 15% of the length of the hand breadth. However, this generalizeddefinition of longitudinal conformance can be employed whether or notthe gesture system uses gesture levels. A finger map consisting of a rowof four contiguous square input regions, with rest locations centered inthe squares, employs a margin of just below 18.4% for the 25-degreerange around the perpendicular to the orientation line. A margin of 20%thus strongly asserts that the input area or input regions shouldgenerally be taller than they are wide, and a margin of 28% asserts thatthe input area or input regions should generally be more than 1.5 timestaller than they are wide. Note that this generalized definition oflongitudinal conformance and the “Nth-level” definition do notnecessarily limit linear directions to the same set of ranges.

In the above criteria for the various definitions of longitudinalconformance, the gesture system need not specify or determine either oflocations R₁ or R₂, and it need not specify or determine the assumedorientation line through locations R₁ or R₂. The gesture system alsoneed not specify or determine either the line perpendicular to theassumed orientation line or the direction angle for this perpendicularline. The gesture system likewise need not specify or determine eitherof linear directions D₁ or D₂, and it need not specify or determine ahand breadth or a percentage length of the hand breadth. The abovecriteria cite these measures for the purpose of describing geometricproperties of linear directions that should be longitudinal for mostpeople's hands.

FIGS. 40A-40B illustrate the criteria for determining longitudinalconformance using margins. FIG. 40A illustrates the criteria for afinger map of input regions 620-623, and FIG. 40B illustrates thecriteria for an input area 624 for which the host provides fingeridentities. A possible orientation line 633 passes through possibleend-of-series rest locations 625 and 628. The dashed lines 668-669passing through the rest locations 625 and 628 correspond to lineardirections D₁ and D₂ in the longitudinal conformance criteria, eachrepresenting a linear direction along which margins are measured. Eachdashed line 668-669 may represent a channel direction for its respectivefinger identity, but each can also represent any linear direction thatis within 25 degrees of a perpendicular 634 to the orientation line 633.Neither figure depicts this latter constraint, but the constraintrequires that the angles θ₁ and θ₂ each be within 25 degrees of angleβ₅. M₁ and M₂ are lengths for the margins for rest locations 625 and628, respectively. Rest location 625 is at least a distance of M₁ fromboundaries of the input area along both directions of the lineardirection that dashed line 668 represents, and rest location 628 is atleast a distance of M₂ from boundaries of the input area along bothdirections of the linear direction that dashed line 669 represents. Thepossible orientation line 633 thus satisfies the margin requirements forits rest locations. Both FIGS. 40A and 40B depict a gestured paththrough locations 635-637 and a distance vector 638 for the gesturedpath. The simple linear direction of the path is the linear direction ofthis distance vector 638. The dashed line 639 represents this simplelinear direction, as does the direction angle ϕ for the vector 638. ϕfalls within 25 degrees of the perpendicular 634 to the orientation line633, whose rest locations 625 and 628 have proper margins withboundaries of the input area, and therefore the path through locations635-637 is considered to have a longitudinally conformant lineardirection.

Unless the device restricts a finger to a specific path, it is likelythat the longitudinal path along which any finger gestures will varyslightly from gesture to gesture. A set of gesture bases shouldtherefore be multi-selectable on at least one set of linearassociations. The definitions of longitudinal conformance do not requirethat fingers be able to gesture both forward and backward and arecompatible with gesture systems that only recognize forward gestures orbackward gestures. On some devices, it may even be that all lineardirections are longitudinally conformant. Consider FIG. 37B, which showsan input area for which the host may provide finger identities. If thehost designates the top and bottom of the input area but allows the handto take any orientation, only the strictly horizontal direction cannotbe longitudinally conformant, provided that the input area is largeenough to accommodate the appropriate margins. If the host does notdesignate top and bottom, it is possible that any orientation of thehand may be valid. Imagine a device that extends the boundaries of theinput area 624 in FIG. 39B to accommodate the entire page, allowing forany orientation of the possible rest locations 625-628. If the devicecan detect any finger identity in any location, any two locations arepotentially the end-of-series rest locations 625 and 628, making alllinear directions longitudinally conformant on this device.

7.10.3.3 Longitudinally Selectable Parameters

A set of gesture bases may be multi-selectable on a set of linearassociations, and a set of linear associations may be longitudinallyconformant with a series of finger identities. When a set of gesturebases is multi-selectable on a set of linear associations that islongitudinally conformant with the series of available finger identitiesof the set of gesture bases, the set of gesture bases represents acollection of gestures that can be input on longitudinal paths whileselecting among multiple values for each gesture parameter of a gesture.When multiple values for a gesture parameter of a gesture can be inputwhile restricting the gesture to longitudinal paths, the gestureparameter is said to be “longitudinally selectable.” This “longitudinalselectability” property of gesture parameters is crucial for the rapidinput of finger-mapped gestures. Gesture parameters have to bespecifically designed for longitudinal selectability, making theselection of gesture parameters available to the gesture system centralto the ability of the gesture system to provide rapid data entry.

Consider FIGS. 8A and 8B, which depict longitudinal gestures selectingamong multiple values for both a gesture level parameter and a gesturedirection parameter. Each of these figures shows a finger map. The dotsin dashed circles 104-107 on each finger map represent finger restlocations, and the lines 108-111 that pass through the rest locations104-107 represent longitudinal paths along which the fingers can travel.The paths depicted happen to be lines. Their linear directions arelongitudinally conformant with the series of finger identities given asthe index, middle, ring, and pinky fingers. FIG. 8A depictssingle-finger strokes, and FIG. 8B depicts two-finger sweeps of adjacentfingers. The solid heavy lines 108-111 in FIG. 8A and the solid heavylines 112-114 in FIG. 8B each indicate the direction and length of agesture as a whole and are referred to as “direction/length” lines.Because the gestures of FIG. 8A are strokes, the direction/length linesof this FIG. 108-111 pass through the rest locations 104-107. Becausethe gestures of FIG. 8B are two-finger sweeps, its direction/lengthlines 112-114 are on the boundaries between the input regions 100-103 ofeach pair of fingers that contributes to a gesture.

FIGS. 8A and 8B both illustrate the longitudinal selectability of thegesture level and gesture direction parameters. The graphical notationshows each gesture selecting among multiple parameter values for each ofthe two gesture parameters. Dotted line segments (e.g. 115-118)partition the direction/length lines at the boundaries between gesturelevel threshold lengths for each direction of movement of the fingers.For example, a finger in the pinky finger input region 103 of FIG. 8Acan select between inputting ‘S’ and ‘T’ by gesturing forward either adistance greater than or a distance less than the distance from the restlocation to the dotted line boundary 115 between the ‘S’ and the ‘T’,respectively, after having initiated the gesture at point 107.Similarly, according to FIG. 8B, if fingers in the middle and ringfinger input regions 101-102 initiate a gesture at points 105-106 andgesture a backward sweep, they select among ‘5’, ‘6’, and ‘7’ accordingto a combined distance of travel calculated for the two fingers,delimited by the dotted lines 117-118. These two figures depict thelengths of gestures according to the direction of travel of thegesturing fingers, regardless of where the fingers start gesturing inthe input regions 100-103. There is no requirement that the fingersstart at the rest locations 104-107, and the distances depicted (e.g.115-118) do not inherently partition the regions along fixed boundaries.

7.10.4 Longitudinal Bias

A gesture system can take advantage of the efficiency of longitudinalgestures without requiring that all gestures be longitudinal, becauseonly the user requests that are best input rapidly should be availableas longitudinal gestures. User requests that can be input less rapidlycan be made available as lateral gestures. Normally the most frequentlyinput user requests are those that should be input rapidly, but it isreasonable to associate even high frequency user requests with lateralgestures when the user requests represent delimitations of the databeing input. Each user request intended for rapid input should onlyrequire gesturing by a single hand, parameterized by finger identitiesfrom a single hand, because it can be difficult for users to coordinatetogether the fingers of different hands. A gesture system thatdisproportionately associates longitudinal gestures with the userrequests that should be input most rapidly is said to exhibit“longitudinal bias.”

The forward and backward directions in which fingers travel may varyfrom hand to hand for a given hand position, so the directions oflongitudinal bias may also vary from hand to hand. Longitudinalconformance, as previously defined, establishes a direction of bias thatshould suffice for the majority of users. In particular, the gesturesystem may establish longitudinal bias by disproportionately associatinguser requests with gestures that traverse paths having simple lineardirections found in a set of linear associations, where the set oflinear associations is longitudinally conformant with a series of fingeridentities of the gesture system, for some definition of “longitudinallyconformant.” These gestures are longitudinal gestures under the selecteddefinition of longitudinal conformance, and they travel longitudinalpaths also established by this definition. The gesture parameterizationsavailable to these longitudinal gestures are called “longitudinalparameterizations.”

The gesture system establishes longitudinal bias by predominantlyassociating longitudinal parameterizations with user requests thatshould be available for rapid input. To make many user requestsavailable for rapid input, the gesture system will have to associatemany user requests with longitudinal gestures. A longitudinal gesture ofany given combination of finger identities can reasonably select from atleast four gesture parameterizations, as illustrated in FIGS. 8A, 8B,61A, and 61B for parameterizations including gesture level and gesturedirection.

Consider a gesture system that inputs characters. Most text that usersenter is made of words consisting of letters, so requests to inputletters should be among those that longitudinal gestures represent.Letters vary from alphabet to alphabet, and alphabets vary in size, withsome alphabets potentially having hundreds of letters that vary by theirdiacritical marks. It is not necessarily pragmatic to have longitudinalgestures for all possible letters, but it is reasonable to have them fora majority of letters of a basic alphabet. A normal distribution of analphabet across both longitudinal and lateral gestures would associateroughly an equal number of letters with each of the two classes ofgesture, because varying direction is a simple and fast way to selectamong letters. To establish a longitudinal bias, longitudinal gestureswould have to represent a significant majority of the letters, so thatlongitudinal parameterizations indicate a significant majority ofletters. What constitutes a significant majority is somewhat arbitrary.It suffices for longitudinal parameterizations to represent at leasttwo-thirds of the alphabet. That is, longitudinal parameterizationsshould indicate requests to input at least two-thirds of the letters ofa basic alphabet. In an alphabet with 26 letters, such as the Englishalphabet, longitudinal parameterizations should indicate at least 18letters. The fewer the number of finger identities, the more significantthis restriction is. In particular, this restriction is significant atfour finger identities or fewer for longitudinal gestures.

When the direction of a gesture is able to select a letter, it seemsreasonable to allocate letters to the four most readily selecteddirections—forward, backward, left, and right. It also seems reasonablethat only after allocating letters to all of these directions would onelook for ways to allocate more than one letter in a given direction. Forexample, only after already allocating letters to the forward, backward,left, and right directions would one consider finding a way to make morethan one letter available forward or backward. Yet, finger-mappedgestures are most efficient in the forward and backward directions,suggesting that there is no need to map letters to left and rightdirections. In particular, it is a surprising feature of finger-mappedgesture systems that it is most efficient to allocate multiple lettersin each of the forward and backward directions before assigning lettersto left and right directions. One way to express this property is to saythat the gesture system assign at least two distinct letters to aforward or backward direction before associating any letter at all witha left or right direction. For example, such a gesture system could havetwo letters available to the forward direction while having no lettersavailable to any direction that is left of the forward and backwarddirections. Similarly, two letters could be available to the forwarddirection while no letters are available to any direction that is rightof the forward and backward directions. The same property is expressedby replacing the forward direction with the backward direction in theseexamples. Several figures in this specification (e.g. FIGS. 35A-35D and36A-36D) show letters being assigned equally to all four directions inorder to provide a simple representation of user requests associatedwith those directions. However, to provide gesturing efficiency, theaccompanying sample implementation only associates letters with theforward and backward directions, as shown in FIGS. 61A-61B. Similarly,FIGS. 8A-8B also illustrate gesturing only in the forward and backwarddirections to input any letter of the English alphabet.

Another approach has longitudinal parameterizations indicating at leasta certain number of the most frequently used letters of a language. Forexample, the sample implementation provided in this specificationassociates the seven most frequent letters of the English language withlongitudinal parameterizations. These letters are A, E, I, N, O, S, andT. These seven letters are actually mapped to seven of thefastest-to-input gestures, which are stroke gestures of gesture level 1.Other languages should also map the most frequently entered letters tothe fastest-to-input finger-mapped gestures. According to the charts inthe Wikipedia entry for “Letter frequency,” among the Romance languagesEnglish, French, German, Italian, Portuguese, and Spanish, the mostfrequently input ten letters are A, D, E, I, L, N, O, R, S, and T. Undera generic rule for Romance languages, longitudinal parameterizationsshould indicate at least seven of these ten letters.

Letters typically come in lower case and upper case forms, but thistreatment of longitudinal bias is independent of letter case. Forclarity, the claims that accompany this specification indicate lettersin upper case form, but as used in the claims, a “letter” selected froma group of letters is any representation of a letter that appears in thegroup. Representations for a letter include its upper case form, itslower case form, and an abstract representation that is independent ofletter case. The letters of a set of letters are “distinct” if, and onlyif, no two letters of the set are representations for the same letter ofthe group from which the letters were selected. In particular, twoletters are “distinct” if, and only if, the letters are notrepresentations for the same letter of the group.

Punctuation symbols can be more frequently input than many letters, soit beneficial to also associate them with longitudinalparameterizations. However, most punctuation symbols serve as delimitersin text, so it may be helpful to break the flow of input by associatingsome of them with lateral gestures, regardless of whether there are alsolongitudinal parameterizations for the lateral gestures. For example,dashes and carriage returns are usefully implemented as right- andleft-moving gestures, respectfully. The space character is typicallymore common than any letter and so should be available as a longitudinalgesture in order to maximize the rate of input, but it is also helpfulto implement the space as a lateral gesture in order to suggest spacingapart adjacent words. There may be a need to balance mnemonic value withthe rate of data input.

7.10.5 Gesture Length Accommodation

User input can be made still more efficient by sizing gesturing areas toaccommodate the patterns of gesturing that each finger is likely toperform. Although finger-mapped gestures can end outside the input areaon touch devices that allow it, there are benefits to improving the oddsthat gestures end in the input area. Consider a user gesturing a fingerforward to input a stroke. If the stroke ends outside the input area,the user must first reposition the finger above the input area again tobegin another gesture with the finger. Assume that the forward gestureinstead ends on the input area and that the finger begins its nextgesture from this location. If the subsequent gesture is a backwardgesture, the finger moves closer to the location where the forwardgesture started. This location is also on the input area, so afterperforming the backward gesture, the finger likely remains above theinput area once again positioned for the next gesture. A gesture systemcan therefore improve gesturing efficiency by sizing the input area toaccommodate the lengths of gestures according to where fingers gestureon the input area.

7.10.5.1 Longitudinal Lengths

The simplest way to accommodate gestures is to ensure that a gesture canstart and end in the input area. This constraint allows a finger torepeatedly gesture in opposite directions by starting each gesture wherethe previous gesture ended. However, the input area should be sized toaccommodate longitudinal gestures, since these are the gestures intendedfor rapid input. In particular, the input area should be sized toaccommodate anticipated gesturing patterns of longitudinal gestures. Agesturing pattern involves a sequence of gestures and the locations atwhich the gesturing fingers start and end the gestures on the input areaduring the sequence. Gesturing forward with a finger, lifting thefinger, placing the finger at the location where it lifted, and thengesturing backward is an example of a longitudinal gesturing pattern.Each gesturing pattern requires a minimum length of input area in whichto perform the pattern. Accommodating longitudinal gesturing patternsthus entails establishing a minimum length of the input area in at leastone longitudinal linear direction. The longitudinal linear directionsare those that are longitudinally conformant with a series of fingeridentities of the gesture system.

When an input area is a finger map, each input region of the finger mapshould accommodate the gesturing patterns of its associated finger.Consider a finger that begins a gesture in an input region, lifts at alocation on the input area, and then inputs a second gesture beginningat the location where the finger lifted. Barring potential fingeridentity reassignment via conflict resolution, the two gestures onlyindicate the same finger identity if the location where the secondgesture begins is in the input region where the first gesture began. Inorder to accommodate the gesturing pattern, the first gesture shouldbegin and end in the same input region. This constraint is sufficient toaccommodate many gesturing patterns, but it does not ensure theaccommodation of longitudinal gesturing patterns. Accommodatinglongitudinal gesturing patterns on input regions requires establishingminimum lengths of the input regions along at least one longitudinallinear direction, possibly a along different longitudinal lineardirection for each input region.

A length of an input area or input region in a longitudinal lineardirection is called a “longitudinal length” of the input area or inputregion. The minimum length that a gesturing pattern requires is calledthe “accommodation length” of the gesturing pattern. Accommodationlengths may vary from finger to finger, because different fingers maygesture over different distances. A gesture system most generallyaccommodates gesturing patterns by ensuring that the longitudinallengths available to finger identities are at least the accommodationlengths of the gesturing patterns for the finger identities. However,the input area need not accommodate the gesturing patterns of fingersfor hands in every possible hand position. Some positions of a hand mayleave a finger with little room for longitudinal gesturing and may beinappropriate for rapid data entry. To accommodate gesturing patterns,the gesture system need only provide one hand position. Even so, themore hand positions there are that accommodate gesturing patterns, theeasier gesturing may be for the user.

FIGS. 39A-39B and 40A-40B provide examples of longitudinal lengths forinput regions and input areas. As previously explained, the dashed lines629-632 and 639 in these figures represent longitudinally conformantlinear directions. The dashed lines 629-632 in FIG. 39A each passthrough a location 625-628 on their respective input regions 620-623 andthus depict lengths of their respective input regions 620-623 alonglongitudinal linear directions. Each of these dashed lines 629-632 cantherefore be considered a longitudinal length of an input region.Similarly, the dashed lines 629-632 representing the same lineardirections each pass through a location 625-628 in the input area 624 ofFIG. 39B and thus can be considered longitudinal lengths of the inputarea 624. In FIGS. 40A-40B, the dashed line 639 represents the lineardirection of a gesture through locations 635-637. According to thesefigures, this linear direction is longitudinally conformant, and so thedashed line 639 through locations 635 and 637 can be considered to be alongitudinal length. In FIG. 40A, dashed line 639 provides alongitudinal length of the input region 621, while in FIG. 40B, itprovides a longitudinal length of the input area 624.

Accommodating gestures entails ensuring that subsequent gestures canstart near where previous gestures end. This requires that eachsubsequent gesture be able to indicate the finger identity of theprevious gesture. Accommodating a gesture that indicates a fingeridentity therefore requires that the gesture be able to start and end inan isogeometric element for the finger identity. In particular, toaccommodate a gesture, each finger of the gesture's selecting finger setmust start and end the gesture in a geometric element that specifies adistinct finger identity of the gesture's selected identity set,although it is possible for one geometric element to specify all of thefinger identities. For example, if the gesture is a stroke, the fingerstarts and ends in a geometric element that specifies the finger'sfinger identity. On a finger map, this geometric element is the inputregion for the finger identity. When the host provides fingeridentities, this geometric element is the entire input area. Similarly,the longitudinal length available to a finger identity is the lengthalong a longitudinal direction of a geometric element that specifiesthat finger identity. Input regions and input areas are the isogeometricelements that have the longitudinal lengths.

7.10.5.2 Accommodation Lengths

The accommodation lengths required for efficient gesturing depend on thetypes of gestures being accommodated. Gesture level and gesturedirection are the most efficient gesture parameters, so it is best forthe gestures that are intended for rapid input to use at least one ofthese two parameters. Moreover, to make these gestures as fast to inputas possible, the gestures should not require the user to performadditional gesturing for additional gesture parameters. If a gestureonly implements gesture direction, the accommodation length can be quiteshort, such as the length of a gesturing threshold distance.Accommodating gesture length is therefore mainly an issue ofaccommodating lengths of the gesture level parameter. When a gesturesystem supports varying gesture level thresholds, the gesture systemshould accommodate at least the shortest gesture level thresholds, sincethese are more amenable to rapid input.

The inventor has found that when repeatedly gesturing a finger in thesame direction, a finger tends to either start at its rest location orslightly opposite the rest location from the direction in which thefinger will gesture. A finger gesturing forward tends to start justbelow (backward of) the rest location, and a finger gesturing backwardtends to start just above (forward of) the rest location. However, whena finger gestures in one direction and then gestures in the oppositedirection, the finger tends to start the second gesture at or near thelocation where it ended the previous gesture. A gesture system thatallows the user to gesture in both forward and backward directionstherefore benefits from ensuring that the input area has a longitudinallength that accommodates the full length of a gesture. In addition tosparing the user from having to carefully target fingers to the inputarea when alternating the direction of successive gestures, thisincreases gesturing speed because a finger only needs to lift brieflybefore touching again at or near the location where it had lifted.

There are a variety of ways to compute the minimum length thataccommodates gestures inputting gesture levels. The inventor estimatesthat when gesturing at two gesture levels, a finger tends to indicategesture level 2 by travelling at least 50% farther than it travels toindicate gesture level 1. FIG. 41A depicts two gestures. The firstgesture 640 has length G₁ and gesture level 1. The second gesture 641has length G₂ and gesture level 2. The second gesture 641 is 50% longerthan the first 640. Assuming that users gesture at gesture levels 1 and2 with these average lengths, users will intend to indicate gesturelevels 1 and 2 even when gesturing at slightly above and below theselengths. The threshold distance for gesture level 2 should therefore behalfway between these two lengths. The figure indicates this level 2threshold distance as L₂, whose value is 1.25G₁. The length L₂ is theminimum length of a level 2 gesture, and the length G₂ is an averagegesture level 2 length, at least for users who strive to minimize thelength of level 2 gestures. Because a gesture level parameter supportsat least two gesture levels, these two lengths are useful for computingvarious possible accommodation lengths for accommodating gesture level.

FIGS. 41B and 41C depict a user gesturing forward with one finger andthen gesturing backward with the same finger starting at the locationwhere the previous gesture lifted from the gesture surface. For example,in FIG. 41B, a finger may gesture along path 642, lift at location 643,touch again at location 643, and gesture in the opposite direction alongpath 644. The length of each gesture in FIG. 41B is the minimum gesturelevel 2 length L₂. In order for the finger to gesture both ways on theinput area without having to re-target the input area, the accommodationlength must therefore be at least L₂ for the finger. This is the minimaluseful accommodation length. It is more practical to assume that theuser gestures at the average level 2 length G₂ and also that the userwill only touch near the location where it previously lifted and notnecessarily exactly at that location. This is the scenario that FIG. 41Cdepicts. The figure assumes that the error in touch location is the sameas the error available to the ending locations of the average gesturelevel 1 and gesture level 2 lengths, as shown in FIG. 41A. This error is¼G₁. The accommodation length for this gesturing pattern is consequently1.6L₂. This gesturing pattern captures the user beginning backwardgestures at the top of the input area and forward gestures at the bottomof the input area, which would require the user to actively aim for thetop or bottom of the input area when the input area only accommodatesthis length. It is possible for the user to do this, but it is notideal.

Fingers tend to start gesturing at their rest locations after a pause ingesturing, and they tend to start gesturing near the rest locations whenrepeatedly gesturing in the same direction. In order to accommodate agesture that follows a gesture that starts near rest locations, theaccommodation length must be at least twice the gesture level 2threshold. FIG. 41D depicts this minimal scenario, illustrating anaccommodation length of 2L₂, which does not give the user room to errorin the length of the gesture or in the placement of the fingers. FIG.41E allows the user room to error in gestures that start near the restlocation, accommodating at least length 2G₂, or 2.4L₂, which is twicethe minimal average length of gesture level 2. This length assumes thatthe user can place the finger closely to where the previous gestureended. Also allowing for placement error around the rest locationsuggests a minimal accommodation length of 2.8L₂. However, FIG. 41E doesnot account for the apparent tendency of fingers to start slightlyopposite the rest location from their intended direction of travel. FIG.41F illustrates this scenario. Excluding placement error, theaccommodation length is once again 2L₂, or twice the length of thegesture level 2 threshold. Including a placement error of ±¼G₁ suggestsa minimal accommodation length of 2.4L₂. Under these computations, theaccommodation length varies from L₂ to 2.8L₂, with the lengths 2L₂ and2.4L₂ each making two appearances among the useful calculations.

Accommodating gesture levels of 3 or greater would entail similarcalculations. It is reasonable to place each average gesture levellength at 50% greater than the average length of the previous gesturelevel. As the number of gesture levels increase, the gesture lengthsthat are required to confidently indicate the gesture levels increaserapidly. In addition to the increased difficulty of learning manygesture lengths, long gestures are not well suited for rapid data input.While there may be benefit to sizing input areas to accommodate gesturelevel 3 gestures, there likely is not much benefit to sizing for gesturelevels 4 and greater. At a minimum, gesture systems employing gesturelevel should accommodate gesture level 2, regardless of how many gesturelevels are available. The longer that the longitudinal lengths availableto gesturing fingers, the more flexibility the user has for error, andhence the faster the user can accurately input data. An accommodationlength of at least 2 cm appears to be sufficient for entering gesturelevel 2 gestures.

7.10.5.3 Accommodation by Design

As with the linear directions of a set of longitudinally conformantlinear associations, a touch device need not compute accommodationlengths to accommodate gesturing patterns. A touch device can insteadaccommodate the lengths of gesturing patterns by design. For example,when the input area has a longitudinal length for some linear directionavailable to a finger of a finger identity and this longitudinal lengthis at least the threshold length of a particular gesture level, thetouch device accommodates a finger of that finger identity inputting agesture of the particular gesture level along that longitudinal lengthand linear direction, regardless of whether the gesture system hasspecified or determined the longitudinal length or the linear direction.Assertions about longitudinal lengths and accommodation lengths conveymeasures of the input area for purposes of establishing geometricproperties of the input area. It is significant and surprising that itis beneficial to size an input area or input region to ensure thatgesture level gestures end in the input area or input region, becausegesture level is based on gesture length and the length of the gestureis independent of whether the gesture ends in the input area or inputregion.

The observation that fingers tend to gesture forward from slightlybackward of their rest locations and backward from slightly forward oftheir rest locations suggests a design under which regions need notindividually accommodate gesture lengths. Consider FIGS. 61A and 61B,which illustrate a mapping of gesture parameterizations to charactersfor the sample implementation included in this specification. The sampleimplementation explains these figures, but for present purposes it needonly be understood that the characters above the dashed line 1000 areinput by gesturing forward and those below the dashed line 1000 areinput by gesturing backward from anywhere within the respective verticalregions 994-997. In the context of the sample implementation, each ofFIGS. 61A and 61B represents a single finger map, but it also ispossible for each figure to represent two distinct finger maps 1001 and1002. One of the two distinct finger maps 1001 consists of the portionsof regions 994-997 that are above the dashed line 1000, and the otherfinger map 1002 consists of the portions of regions 994-997 that arebelow the dashed line 1000. The characters available to the finger mapabove the dashed line 1000 are those listed in the finger map below thedashed line 1000, and vice versa. For example, to input the letter ‘a’the user starts the gesture with the index finger in the leftmost region994 of the top finger map 1001 and gestures a short stroke downward.Because the height available to the fingers is now split across twofinger maps 1001-1002, the two finger maps together can be designed toaccommodate gesture lengths rather than each finger map individually.The main drawback to this approach is that the user must accuratelylearn to have fingers start gestures in the appropriate finger map. Thisdrawback can be avoided by making the gestures available to the top andbottom finger maps identical, but then both finger maps would be actingtogether as a single finger map and not as two finger maps.

7.11 Calibration Gestures

Of the many ways to input a hand calibration, calibration gestures areparticularly convenient. A “calibration gesture” is a gesture that theuser performs with the hand on the gesture surface to specify propertiesof a new hand calibration directly from measurements of the hand. Acalibration gesture can specify a new hand calibration from scratch, orit can specify adjustments to a previous hand calibration. In the lattercase, the calibration gesture may act upon an existing finger map,employing the finger map to determine the identities of some or all ofthe fingers of the calibration gesture. Because a hand calibrationpotentially specifies the position and orientation of a hand in additionto the relative characteristics of a hand, calibration gestures includegestures that reposition and reorient finger maps.

7.11.1 The Calibrating System

A calibration gesture may either position an input area 8 on theavailable gesture surface or position it more locally on a providedsurface allocation. When a calibration gesture positions an input areaon the available gesture surface, it selects the position of the surfaceallocation. A gesture system 7 as defined in this specification does notdo this, so a gesture detection system of the host 4 called a “surfacemonitor” 5 implements the gesture instead. When a calibration gestureonly positions an input area on a provided surface allocation, afinger-mapped gesture system may implement the gesture. The entity thatimplements calibration gestures—whether it is the surface monitor or afinger-mapped gesture system—is called the “calibrating system” 6.

The surface monitor is responsible for receiving calibration gesturesthat position an input area on the gesture surface available to a touchdevice. The host acquires a hand calibration from the calibrationgesture, establishes an appropriate surface allocation, and instantiatesa gesture session on that surface allocation using the hand calibration.The hand calibration specifies enough information to define a fingermap, so the host can optionally constrain the surface allocation to justthe input area that the finger map requires, possibly allocating theremaining area for other purposes. The gesture system then establishes afinger map in accordance with the hand calibration.

The gesture system may provide services to the host that allow thesurface monitor to input a calibration gesture that leverages itsexisting finger map. If the surface monitor allows users to inputcalibration gestures on top of established input areas, it must eitherpreclude gesture conflicts by the design of the calibration andfinger-mapped gestures allowed or else provide a means for resolvingconflicts. Regardless, calibration gestures are ultimately delegated toa particular calibrating system for input and processing.

The calibrating system inputs a calibration gesture in order to acquirea hand calibration. The hand calibration of a calibration gesturecharacterizes a correspondence series of at least two non-thumb fingersof a particular hand. This correspondence series is called the “fullnon-thumb series” of the hand calibration. For example, a handcalibration that includes all four fingers and optionally the thumbspecifies a full non-thumb series of four fingers corresponding tonon-thumb finger identities. The calibration gesture should alsoindicate whether it is for the left or right hand, so that the hands canbe characterized separately. If a calibration gesture is performed forone hand and a hand calibration is not yet available for the other hand,the gesture system can default both hands to the newly input handcalibration. Distinguishing the hand facilitates allowing differenthands to input the same user requests or different user requests to beavailable to different hands.

Some calibration gestures specify more information than others. Forexample, one calibration gesture may only provide finger rest locations,while another also provides finger forward directions. When the userinputs a partial hand calibration, the calibrating system can assumedefaults for the unspecified values, or if possible, estimate remainingvalues from the values that are provided. If the calibrating systemalready has a pre-existing hand calibration for a hand, it can merge thepre-existing hand calibration with the new hand calibration. Thecalibrating system may even maintain different hand calibrations for theleft and right hands. If the calibrating system maintains handcalibrations for multiple users, it can guess the user by finding thepre-existing hand calibration that most closely approximates the newlyentered hand calibration, though the user should be given a chance toconfirm the guess. To indicate or confirm the user performing acalibration gesture, it may be best for a multi-user system to requireusers to input an identifier or password immediately after performingthe gesture. The calibrating system may also attempt to recognize theuser from the characteristics of the calibration gesture itself.

7.11.2 Two-Finger Calibration Gestures

The simplest calibration gesture is a placement of two fingers of thefull non-thumb series on the gesture surface. The identities of the twofingers must be known in advance, so that the calibrating system caninfer the locations of the remaining fingers, whether by interpolationor by extrapolation or both. The hand calibration is most reliable whenthe locations are for the end-of-series fingers and the interveningfingers are interpolated.

Assume the ideal case that the finger identities are for theend-of-series fingers of the full non-thumb series. The end-of-serieslocations are sufficient to specify the spacing and orientation of thehand calibration, but they are not sufficient to specify whether thehand calibration is for the left or right hand or to distinguish theforward or backward directions of finger movement. The moststraightforward way to distinguish the forward and backward directionsis to either hardcode the distinctions or to receive the informationfrom the touch device. Most smartphones and tablets automaticallyreorient top and bottom as the user orients the device, so it isreasonable for the gesture system to designate the forward and backwarddirections as the device indicates. When top and bottom are known for adevice, so are left and right.

On devices for which left and right are known, the calibration gesturecan indicate whether it is for the left or right hand via an asymmetryin the gesture. One such calibration gesture is the “hand selectionslide.” To perform a hand selection slide, the user places on thegesture surface the two fingers that the calibration gesture requiresand moves them as a unit a distance in a direction. The direction thatthe fingers move indicates whether it is for the left or right hand. Forexample, moving left might indicate the a hand calibration for left handand moving right might indicate a hand calibration for the right hand.In order to allow this gesture to be performed on top of an input areathat allows sweeps, either the gesture system must not recognizetwo-fingered sweeps directed left or right, or the fingers of the handselection slide must be spaced apart at least a certain distance greaterthan the distance that the gesture system allows for the fingers of asweep. Otherwise the calibration gesture might be interpreted as asweep, or vice versa.

Another way to indicate the left or right hand is determine it from thelast of the two fingers to touch. For example, if the finger on the leftis the last finger to touch, the hand calibration could be for the righthand, and it could be for the left hand otherwise. However, a gesturethat only involves touching two fingers without moving them is hard todistinguish from accidental touches or intentional attempts to selectuser interface components.

7.11.3 The Drumming Gesture

A more general and more natural gesture that uses the order in whichfingers touch is called “drumming.” To perform a drumming gesture—to“drum”—the user strives to touch each finger of the full non-thumbseries to the gesture surface, while being sure that a finger near adesignated end of the series is among the first or last fingers totouch, depending on the calibrating system. The calibrating systemestablishes a correspondence series for the fingers according to therelative positions of the finger touches. For example, if the lastfinger to touch must be closer to the index finger than the pinkyfinger, the user strives to drum the fingers starting with the pinkyfinger and ending with the index finger, and the last finger identifiesthe index finger and hence all of the remaining fingers according totheir locations relative to the index finger. The touch locations mayalso serve as rest locations for the fingers in a hand calibration.

7.11.3.1 Recognizing Drumming

The calibrating system recognizes a drumming gesture by attempting tointerpret a hand calibration from a series of finger touches that couldbe a drumming gesture. The calibrating system detects a sequence offinger touches, attempts to order them into a series according to theirrelative locations on the gesture surface, and attempts to identify oneof the end-of-series fingers as having a particular finger identitybased on the temporal order in which the fingers touched. Having anorder for the finger touches and having the identity of a finger at oneend of the series together establishes the finger identities for theremaining finger touches, therefore establishing a correspondence seriesfor the finger touches and their fingers. A drumming gesture can spanmultiple facets, as long as the relative placement of the facets isknown so that a correspondence series can be established.

Assume that the last finger to touch identifies the end-of-series fingernearest to it. For example, the last finger to touch may indicate thatthe nearest end-of-series finger to it is the index finger. The lastfinger to touch may even identify itself as the index finger. The userdrums the fingers starting with fingers near the pinky finger and endingwith fingers near the index finger. This information is not sufficientto indicate whether the user drummed with the left hand or the righthand, but given a left-and-right sense for the gesture surface, theorder determined for the finger identities does indicate the hand usedand hence which hand is being calibrated.

When the gesture surface has designated left and right sides and theleft-to-right direction is parallel to a coordinate axis, the moststraightforward way to determine an order for the correspondence seriesis to sort touch locations of the fingers by their coordinate values forthat axis. This approach works when the knuckles of the hand areparallel or nearly parallel with this axis, but consider that whenholding a tablet with the left hand, one orients the tablet relative toone's eyes and body, placing the right hand at rest on the tablet withthe knuckles at an angle relative to the edge of the tablet. In thisorientation, the hand is presumably at an angle relative to theleft-to-right axis. Assuming that the left-to-right axis is the x-axis,adjacent fingers need not have sequential x-coordinate values if thefingers are not touching at locations along a straight line, such aswhen some fingers are positioned ahead of others. Hence, while it issometimes possible to sort the fingers into a series strictly by acoordinate value, the approach is not very reliable.

A more reliable approach to ordering the fingers into a series sorts thefingers by the relative distances between their touch locations. Underthis approach, the calibrating system calculates the distance betweenthe touch locations of every two fingers of the drumming gesture. Thepair of fingers that are farthest apart are designated the end-of-seriesfingers. The calibrating system chooses one of the end-of-seriesfingers, and the remaining fingers are sorted in the series according totheir distances from the designated end-of-series finger, with thefingers nearer to the designated finger being sorted nearer in theseries to the designated finger.

Once the fingers have been ordered into a series, the calibrating systemassigns them finger identities. Assume that the finger identities areassigned according to the last finger to touch. When there are onlythree fingers in the series, an end-of-series finger must be the last totouch because the middle of the three is not nearer to one end of theseries, according to the order of fingers within the series. When thereare four fingers in the series, any finger can be the last to touchbecause each finger is closer to one end of the series. The finger thatis last to touch indicates its near end of the series. The nearness of afinger to an end-of-series finger can be expressed in terms of thenumber of fingers within the series between the finger and theend-of-series finger, where a number of fingers equal to zero indicatesthat the finger is one of the end-of-series fingers. The gesture systemdesignates the nearest end-of-series finger as representing a particularfinger identity and assigns finger identities to the remaining fingersof the series in finger identity order. For example, the gesture systemmay require that the user drum four fingers so that the last finger totouch is closer to the index finger than the pinky finger. When the userdrums the fingers, the last finger to touch then identifies theend-of-series finger nearest to it in the series as the index finger.When the last finger to touch is an end-of-series finger, theend-of-series finger “closest” to it is itself. Once the index finger isassigned, the remaining fingers are assigned in order as the middlefinger, the ring finger, and the pinky finger. The series is now acorrespondence series.

A number of options are available to the drumming gesture. It is anoption for the calibrating system to instead identify an end-of-seriesfinger from the first finger to touch, from the second finger to touch,or from the second-to-last finger to touch. The finger that identifiesthe end-of-series finger is called the “indicator finger.” Thecalibrating system may also require combinations of the touches to occurin a particular order before recognizing the gesture as a valid drumminggesture, such as requiring all of the fingers of the correspondenceseries to touch in the order of their associated finger identities. Thislatter constraint maximizes the probability that the calibrating systemrecognizes a drumming gesture with finger identities in the order thatthe user intended, but the constraint is onerous and slows the user'sperformance of the gesture. A more pragmatic way for the calibratingsystem to ensure that it has received the intended drumming gesture isto require that the indicator finger be an end-of-series finger. In thiscase, finger identities are determined according to either the firstfinger or the last finger to touch during the drumming gesture. Usingthe last finger to touch, whether or not it is an end-of-series finger,is a good compromise between accurately detecting the drumming gestureand giving the user some room to error in the process of attempting tomake the last finger an end-of-series finger. The inventor has alsofound it most natural to drum starting on the pinky finger side andending on the index finger side.

Once the correspondence series is determined, the calibrating system canidentify the drumming hand as either the left hand or the right hand.The most reliable approach to identifying the hand assumes that thegesture surface has designated left and right sides. In this case, thereis a leftmost end-of-series finger and a rightmost end-of-series finger.If the leftmost end-of-series finger identity is nearer to (or identicalto) the pinky finger than the rightmost finger, the hand is the lefthand. If the leftmost end-of-series finger identity is nearer to (oridentical) to the thumb than the rightmost finger, the hand is the righthand. For example, if the gesture system uses four gesturing fingers andthe leftmost end-of-series finger is the index finger, the hand is theright hand.

If the gesture surface does not have designated left and right sides,then absent additional information, the calibrating system does not knowwhich direction on the gesture surface the hand is facing. A left handdrumming on the gesture surface may be indistinguishable from a righthand drumming with fingers aimed in the opposite direction. One possibleway to identify the hand under these circumstances is to assume thatfingers central on the hand touch at locations more in the forwarddirection than the end-of-series fingers. This distinguishes the forwardand backward directions of finger movement. This information togetherwith the order of the finger identities determines the left or the righthand. For example, if the forward direction is in the direction of thenegative y-axis and relative to this forward direction the rightmostend-of-series finger identity is the index finger, then the hand is theleft hand. A simple way to determine the forward direction for a seriesof fingers is to count the finger touches on each side of a line thatcoincides with locations of the end-of-series fingers. The side of theline with the most touches represents the forward direction. However,this approach is unreliable in gestures systems that support gesturedirection because in these systems the ideal finger mapping postureplaces all of the fingers along a line, perhaps inclining the user todrum the fingers roughly along a line.

The calibrating system recognizes a drumming gesture when both (a) thenumber of fingers touching is at least the required number of drummingfingers and (b) a subset of the fingers that are touching satisfies theconstraints of the drumming gesture. When the constraints are lax suchthat fingers touching in any order can indicate a drumming gesture forone of the hands, the drumming gesture is likely incompatible with othergestures having an equal number of finger touches. Lax constraints canalso preclude an anchor finger from touching during the drumminggesture. In particular, the user may want to provide a thumb anchor withthe drumming gesture in order to reliably anchor the hand to the fingermap that results from the drumming gesture. Constraints are not neededto allow an anchor thumb to touch after the drumming gesture hascompleted, but constraints are needed to allow the thumb to touchimmediately prior to the gesture or during the gesture. The followingtwo constraints are together sufficient to allow the thumb to toucheither (a) immediately prior to the gesture or (b) after the firstfinger of the drumming gesture touches but before the second finger ofthe drumming gesture does:

-   -   1) The end-of-series fingers of the correspondence series cannot        be immediately sequential in time. More specifically, a finger        of the correspondence series must touch after the finger at one        end of the correspondence series and before the finger at the        other end.    -   2) The first and last fingers of the correspondence series to        touch must be on opposite sides of the correspondence series.        Hence, if there are only three fingers in the series, they must        touch in order, and if there are four fingers in the series, the        first and last fingers to touch cannot be adjacent if either of        them is an end-of-series finger.

These constraints also allow the calibrating system to properlyrecognize the drumming gesture for most of the combinations of fingertouches when the thumb touches after the second finger of the drumminggesture to touch but before the third finger to touch. These constraintstogether eliminate some combinations of finger touches that mightotherwise be recognized as drumming gestures, but these othercombinations are unlikely. It may actually benefit the user to enforcethe above constraints regardless of whether there is a need to support athumb anchor, in order to reduce the incidence of improperly recognizingdrumming gestures. Note that these constraints are independent of theparticular finger that the calibrating system uses to determine anend-of-series finger identity for the purpose of assigning fingeridentities.

It may also be important for the calibrating system to distinguish thedrumming gesture from finger mapped gestures. If none of the fingers ofthe drumming gesture is allowed to exceed a gesturing threshold, or ifthe drumming gesture requires more fingers than are allowed in a sweepgesture, the drumming gesture can be performed without conflict on topof an existing finger map as a way of recalibrating and repositioningthe finger map.

Once the calibrating system has finger identities for the drummingfingers, it can assign rest locations to the finger identities. Theassignment that is most adapted to the user's hand is to set the restlocation for a finger identity to a location of the finger having thatfinger identity in the drumming gesture. FIGS. 46A-46F, explainedpreviously, illustrate ways to assign rest locations with fewer fingers,such as by extrapolating or interpolating from two fingers. The gesturesystem creates a finger map from these rest locations, thus calibratingthe finger map.

FIG. 48 illustrates the steps for processing a drumming calibrationgesture in the context of receiving finger-mapped gestures. In step 825,the calibrating system first detects the required number of drummingfingers. If the drumming is being performed on top of an existing fingermap, the gesture system may distinguish drumming fingers from fingers ofa selecting finger set by ensuring that drumming fingers are notgesturing. In step 826, the drumming fingers are ordered into a seriesaccording to the relative positions of their touch locations. Thecalibrating system requires that the user indicate the sense of the handby completing the drumming either closer to the index finger or to thepinky finger. The sense of the hand is determined according to anindicator finger, such as the first finger to touch or the last fingerto touch. The calibrating system determines the indicator finger in step827. In step 828, the indicator finger establishes the finger identitiesof the end-of-series fingers nearest and farthest from it; identifyingone end-of-series finger implicitly identifies the other, so this isequivalent to identifying one of the end-of-series fingers. Thisinformation allows the gesture system to associate each drumming fingerwith a finger identity. Finally, in step 829, the calibrating systemprepares an input area of geometric elements that are spaced so that thedrumming fingers have each touched its own input region, with eachgeometric element corresponding to the finger identity associated withthe drumming finger that touched its input region. The gesture systemmay subsequently receive gestures on this finger map.

7.11.3.2 Examples of Drumming

FIGS. 49A-49H illustrate two examples of inputting the drumming gesturewith four fingers on a gesture surface having left and right sides,where the last finger to touch must be the index finger. Each of theeight figures depicts the gesture surface of a touch device. Dots838-847 represent the locations of finger touches, numbered in the orderin which fingers touch the locations except as otherwise described.Assume that the x-axis is the horizontal and increases from left toright. The gesture surface starts as shown in FIG. 49A, with no fingermap being active yet. In FIG. 49B, three fingers touch the gesturesurface in any order at locations 838-840. In FIG. 49C, a fourth fingertouches the gesture surface at location 841. Among all the locations838-841, the distance between locations 840 and 841 is greatest, makingthe fingers at these locations end-of-series fingers. The finger atlocation 841 is therefore an end-of-series finger that was last to touchto one side of three other fingers, making the gesture a drumminggesture. Because location 841 has a lower x-coordinate value thanlocation 840, it is to the left of the other fingers, making it theindex finger of the right hand. The drumming gesture therefore specifieda hand calibration for the right hand. Had location 841 landed to theright of locations 838-840, the drumming gesture would have specified ahand calibration for the left hand. The host allocates a surfaceallocation to the gesture system, which generates the depicted fingermap of regions 830-833. The finger map depicts each finger location838-841 as a rest location centered in its associated input region830-833 and is now available for gesture input. The drumming gestureonly provides the rest locations 838-841 for fingers in the handcalibration; the shapes and forward directions of the input regions830-833 are borrowed from information provided by a previous handcalibration. In FIG. 49D, the right thumb touches at point 842 with noeffect, because this system calibrates after four fingers have touched,making the calibration complete prior to the thumb touching.

The second example of drumming in this sequence begins with FIG. 49E.The fingers that produced the finger map have lifted, but the finger mapremains behind as input regions 830-833 and is available forfinger-mapped gesturing. In FIG. 49F, a finger touches at location 843.This finger happens to be the right thumb, but the calibrating systemdoes not know this yet. Fingers 844, 845, and 846 touch in FIG. 49G,some on the finger map, some off, in any order, except that therightmost finger at location 845 is not the last to touch. Had location845 been the last to touch, the gesture would have indicated a handcalibration for the left hand analogously to the right hand gesture ofFIG. 49C. Because a finger between the farthest separated locations wasthe last to touch, the gesture monitor does not yet recognize a gesture.However, in FIG. 49H, a fifth finger touches at location 847. Thisfinger is an end-of-series finger of the group of four fingers atlocations 844-847 and is farthest from location 845 in the group, withthree fingers on one side of it in the series. The calibrating systemtherefore concludes that the gesture is a drumming gesture. Because thex-coordinate of location 847 is less than that of location 845, thegesture again specifies a hand calibration for the right hand. Thecalibrating system terminates the gesture session of the previous fingermap and starts a new gesture session for the new hand calibration usinga new finger map with input regions 834-837. The new finger map has anew orientation, a new position, and a slightly smaller size than theprevious finger map. In addition, the rest locations are in slightlydifferent relative positions, changing the shapes and positions of theinput regions 834-837. Five fingers were input by the time thefour-finger drumming gesture was recognized, so the finger at location843 is assumed to be the thumb and remains ignored.

FIGS. 50A-50H also illustrate two examples of using the drumming gesturewith four fingers on a gesture surface with left and right sides, but inthese examples, the last finger to touch must be either the index fingeror the middle finger. Moreover, the first and last fingers to touch mustbe on opposite sides of the correspondence series. The locations touched850-859 are the same as those in FIG. 49A-49H, but they touch in adifferent order and are numbered accordingly. The gesture surface againstarts without a finger map, as illustrated in FIG. 50A. In FIG. 50B,three fingers touch the gesture surface in any order at locations850-852. By FIG. 50C, four fingers have touched in series. The lastfinger touched at location 853, thereby indicating that theend-of-series finger nearest to it is the index finger. That nearestfinger has location 852. The fingers at locations 853, 851, and 850 aretherefore designated the middle, ring, and pinky fingers, respectively,and the hand is again designated the right hand. The drumming gesture isnow complete, and a finger map with regions 830-833 is available foruse.

While the finger map of FIG. 50C is in place, the thumb touches atlocation 854 in FIG. 50D. The finger map remains unchanged. However, thethumb lifts in FIG. 50E and touches at location 855 in FIG. 50F.Additional fingers proceed to touch at locations 856-858. By FIG. 50G,four fingers are touching, with the finger at location 858 designatingthe finger at location 855 as an end-of-series finger. The calibratingsystem orders the fingers into a series, but it finds that the firstfinger to touch (at location 855) and the last finger to touch (atlocation 858) are on the same side of the series, so the calibratingsystem declines to recognize a drumming gesture yet. In FIG. 50H,another finger touches at location 859. The number of fingers touchingis at least the minimum required of a drumming gesture, so thecalibrating system again attempts to recognize a drumming gesture. Twocorrespondence series are possible, one for the fingers at locations855, 858, 859, and 856, and one for the fingers at locations 858, 859,856, and 857. The former series does not satisfy the drummingconstraint, but the latter series does. The calibrating system thereforerecognizes the latter series as having performed a drumming gesture.

7.11.4 The Squeeze Gesture

If the device does not indicate left and right, or if the user is to beallowed to specify hand calibrations at arbitrary orientations, thecalibrating system must receive additional input that specifies adirection indicating which way is left or right, top or bottom. Normallythis entails asking the user to do something extra. For example, thefingers could drum to indicate the left or right hand, and then beforelifting, drag all at once in the same direction to indicate the forwarddirection. This gesture is cumbersome by virtue of requiring twodistinct steps. In an ideal gesture, the user should be able to approacha touch device from an arbitrary angle and specify a hand calibration ina gesture with a single motion that amounts to little more than touchingthe gesture surface. Fortunately, there is such a gesture: the “squeeze”calibration gesture. The squeeze gesture is a natural way to provide allof the information necessary for specifying a hand calibration onarbitrary gesture surfaces. In particular, it provides a “handdistinction” to distinguish the left and right hands and enables thecalibrating system to determine finger identities and finger forwarddirections.

7.11.4.1 Recognizing a Squeeze

A user performs a squeeze gesture by placing the thumb and non-thumbfingers on the gesture surface, placing them in any temporal order, butplacing them so that the hand is slightly open. The non-thumb fingersare slightly more apart from the thumb than one would have them fordrumming. The squeeze gesture should include each finger that canparticipate in a finger-mapped gesture. The user then performs a quick,short squeezing action with the touching fingers while they touch,bringing the non-thumb fingers closer to the thumb and the thumb closerto the non-thumb fingers, although not necessarily moving thumb andnon-thumb fingers directly towards each other. The user finishes thesqueezing action with the fingers at their preferred rest locations.This gesture feels like a single squeezing motion that just happens tobe done while touching the gesture surface, as if gently grabbingsomething insubstantial.

FIG. 52A shows a hand performing a squeeze gesture, with arrows (e.g.arrows 865-866) exemplifying the directions in which the fingers maymove. The direction angles of adjacent non-thumb fingers are roughlyparallel. In the figure, these are angles θ_(A1), θ_(B1), θ_(C1), andθ_(D1). The direction angle of the thumb θ_(E1) can reasonably beexpected to differ from the direction angles of the other fingers by atleast π/3 radians by class equivalence. A calibrating system thatdetects five fingers gesturing in this arrangement can reasonably assumethat the gesture is a squeeze gesture. It is possible that the user maynot move some of the fingers enough during the gesture for them to beconsidered intentionally moving. It is also possible that the user willfail to touch some of the fingers during the gesture. As long as thethumb is not among the missing fingers, the calibrating system may stillbe able to identify the gesture as a squeeze. It is sufficient toregister three fingers moving in this arrangement, provided that atleast one of the fingers is the thumb. Even so, the more fingersregistered as moving, the more accurately the gesture is detected, andit may be unintuitive for a user to perform the squeeze gesture withoutinvolving all of the fingers of a hand anyway. It is also possible toperform a squeeze gesture on multi-faceted devices, and most readily ontwo-faceted devices, such as those of FIGS. 4A-4D. To reliably identifya gesture that spans device facets, the calibrating system should knowthe relative positions of the facets.

The squeeze gesture involves fingers moving over a distance, so thecalibrating system should employ a gesturing threshold to distinguishaccidental finger movement from intentional finger movement. In order todistinguish the squeeze from other gestures, the calibrating systemshould establish a minimum number of fingers that must simultaneouslytouch the gesture surface after having exceeded a gesturing threshold.After the calibrating system detects these fingers, the user signalscompletion of the squeeze gesture in any of myriad possible ways. Forexample, the user can signal completion by lifting at least one of thegesturing fingers, by subsequently tapping at least one of the gesturingfingers, or by holding the gesturing fingers relatively stationary for atimeout period, with none of the fingers travelling more than a maximumdistance during this period. Upon detecting the completion of thesqueeze gesture, the calibrating system has a “start location” and an“end location” for each finger. A finger's start location is a locationrepresentative of the start of the gesture, such as the finger's initialtouch location, such as the location at which the finger begangesturing, or such as the location that the finger had the instant whenall of the fingers of the gesture were first detected to besimultaneously touching the gesture surface. A finger's end location isa location representative of the end of the gesture, such as thelocation of the finger at the time the gesture completed. For example,FIG. 53B depicts the actions of fingers inputting a squeeze gesture,with locations 875-879 depicting finger start locations and locations880-884 depicting finger end locations.

If the calibrating system does not recognize any other gesturesinvolving the number of fingers required for the squeeze gesture, it canrecognize the squeeze gesture by the number of fingers alone. However,it is best to look for additional characteristics of the squeeze gestureanyway, in order to eliminate accidental user input or improper squeezegestures that would yield an incorrect hand calibration. The most basiccharacteristic of the squeeze gesture is that it involves a thumb and acorrespondence series of non-thumb fingers. If the thumb can reasonablybe distinguished from a series of other fingers, the gesture likelyconsists of the fingers of a single hand. The following are sometechniques for distinguishing the thumb:

-   -   1) The calibrating system produces a sum for each gesturing        finger by adding up all the distances between that finger's        start location and the start locations for the other gesturing        fingers. The finger with the greatest sum can be identified as        the thumb. To confirm that this finger is the thumb, its sum can        be tested to ensure that it is greater than the next greatest        sum by at least a certain percentage. Alternatively, the sum can        be greater than the average of the sums of the remaining        fingers. The thumb can also be identified from the end locations        of the fingers or from some combination of start and end        locations, but because the fingers are farthest apart at the        start locations, with the thumb well separated from the index        finger, the start locations are the most reliable.    -   2) The thumb can be identified as the finger having the greatest        distance from any adjacent finger, according to the start        locations. To confirm that this finger is the thumb, the        distances between adjacent fingers can be compared, and the        thumb can be required to have an adjacent distance that is        greater than the adjacent distances between other fingers by at        least a certain percentage. It is also an option to compare the        adjacent distance of a potential thumb to the average of the        adjacent distances for the remaining fingers.    -   3) The thumb can also be identified as the outlier finger after        finding a best fit to a line for a number of finger locations        equal to one less than the number of gesturing fingers. The        calibrating system can use either start locations or end        locations. Start locations benefit reliability because the thumb        is farther from the other fingers, but end locations also        benefit reliability because the non-thumb fingers tend to line        up at the end of the squeeze. To confirm that the finger is a        thumb, the potential thumb can be tested for a distance from the        fitted line that is greater than the maximum or average distance        of the remaining fingers to the fitted line. Comparisons can        also be made with a standard deviation.    -   4) Yet another way to identify the thumb is to assume that the        distance vectors of the non-thumb fingers can be ordered into a        series for which adjacent distance vectors have angles differing        by less than π/4 radians, π/6 radians, or some other suitably        small angle. After identifying the non-thumb fingers, the        remaining finger is the thumb. To increase confidence that the        resulting finger is the thumb, the calibrating system can        require that the angle for the distance vector of the potential        thumb be different from the distance vector angles of the        remaining fingers by at least a certain amount by class        equivalence. π/3 radians is a reasonable minimum difference.

If the host provides a finger identity for the thumb, the calibratingsystem does not need to perform any of the above tests. If the host doesnot identify the thumb but the calibrating system is aware that it hasreceived a valid squeeze gesture, the calibrating system can stillemploy any of these techniques for distinguishing the thumb from theremaining fingers, even if it does not test to confirm the presence ofthe thumb. It is also possible to identify the thumb from a combinationof the above procedures or from like procedures. Determining that athumb is present with the proper number of fingers increases confidencethat the gesture is a squeeze gesture.

The calibrating system can also increase confidence that it has receiveda squeeze gesture by checking the non-thumb fingers to see whether theycan be ordered into a series according to locations for the fingersduring the gesture. The calibrating system can also confirm that thethumb moves towards the hand during the squeeze gesture instead of awayfrom the hand. The location of the hand can be given by a “handlocation.” The hand location is any location that represents theposition of the palm or the series of non-thumb fingers. It can be alocation for any non-thumb finger, but if the hand location is too closeto the thumb, the test may be uninformative or provide a false result.It's better to use a location for a finger that is at a distance fromthe thumb or to use an average of locations for several non-thumbfingers. If the thumb's end location is closer to the hand location thanthe thumb's start location, the thumb has moved towards the hand.

7.11.4.2 Squeeze Calibration

A user performs a squeeze gesture in order to input a hand calibration.In particular, the squeeze gesture can identify the fingers, establishfinger rest locations, determine a forward direction for each finger,and indicate the left or right hand. A value that identifies the lefthand or the right hand is called a “left/right hand identity.” Thevalues determined for the hand calibration from a squeeze gesture dependon the requirements of the calibrating system.

At a minimum, the hand calibration depends on identifying one of thefingers as the thumb. If the host does not directly identify the thumb,the calibrating system can identify the thumb via one of the techniquesdescribed above for recognizing a squeeze gesture. The hand calibrationmay also require finger identities for the non-thumb fingers. If theseidentities are not given, the calibrating system can identify the indexfinger as the finger nearest to the thumb according to the finger startlocations. Even if the squeeze gesture does not employ an index finger,it suffices to identify the finger nearest the thumb as the index fingerfor purposes of this discussion. The index finger can also be identifiedfrom the finger end locations, but the identification is most reliablefrom start locations. The calibrating system sorts the non-thumb fingersinto a series by their distances from the index finger or thumb,according to either their start or end locations, placing the indexfinger first in the series. The end locations may be more reliable fordetermining the order of the fingers because the squeezing action bringsthe fingers more in line.

Once the finger identities are known, the calibrating system canassociate rest locations with the finger identities, such as byassigning the rest location for a finger identity to the end locationfor the finger having that identity. It is also an option to set therest location of a finger to the midpoint between the finger's start andend locations, or even to designate a vector for the path of each fingerof the squeeze gesture as a channel line segment. As previouslyexplained for FIGS. 46A-46F, the calibrating system can also infer restlocations using fewer finger locations than finger identities.

In addition, the calibrating system can assign forward directions toeach of the fingers. Being adjacent on a hand, the non-thumb fingersshould all have travelled approximately in the same direction. Eachnon-thumb finger can be assigned a direction angle equaling the angle ofthe vector that starts at the finger's start location and ends at thefinger's end location. These angles each indicate a backward directionfor the finger, and the direction angle π radians different from eachbackward direction provides a forward angle for the finger, indicatingits forward direction. The calibrating system can include these forwardangles in the hand calibration.

A squeeze gesture can also specify the orientation of the hand. Onemodel for specifying the orientation is based on the observation that ahand can be oriented in a particular direction on a tablet so that thefingers generally extend towards a specified “top” of the tablet. Ananalogous “top” direction for a squeeze gesture is called a “generaldirection of finger extension.” There are multiple ways to select ageneral direction of finger extension. The forward angle of anynon-thumb finger is a direction of finger extension for the finger andhence an estimate for the general direction of finger extension for thehand as a whole. The nearest average of the forward angles of thenon-thumb fingers provides a more representative general direction forfinger extension. It is not actually necessary to employ forwarddirections to estimate a general direction of finger extension for thehand, though. The user could be required to touch the thumb on one sideof a line that a set of non-thumb finger locations approximate, and thedirection of finger extension could then be the direction from the thumblocation to any point on the line, such as the nearest point. An anglefor the general direction of finger extension is called a “generalextension angle.”

The hand calibration may optionally include a left/right hand identityas well. There are several ways to make this determination, depending onthe constraints of the calibrating system, but it can be done from therelative location of the thumb. A thumb that occurs to the left or rightof a hand, according to a general extension angle, belongs to the righthand or the left hand, respectively. The hand identity can be determinedby whether a location for the thumb occurs to the left or the right of a“general extension line.” This is a line that has an angle equal to thegeneral extension angle and that passes through a hand location. Thegeneral extension angle indicates the forward direction for purposes ofdetermining which side is left and which side is right.

It is possible to mathematically determine the left/right hand identity.Imagine that the general extension line splits the plane of the gesturesurface into two half-planes. The thumb ends up in the left half-planeor in the right half-plane, thus indicating the right or left hand,respectively. The “thumb vector” is the vector from a chosen handlocation to a location for the thumb. The left or right identity of thehand can be determined by rotating the plane of the gesture surfacearound the chosen hand location through an angle equal to the negativeof the general extension angle. The direction angle of the rotated thumbvector is now class equivalent to the subtraction of the generalextension angle from the thumb vector's original direction angle. If thedirection angle of the rotated thumb vector is negative, the hand is theleft hand. Otherwise the hand is the right hand.

FIGS. 53A-53D illustrate a process for determining a hand calibrationfrom a squeeze gesture. In FIG. 53A, five fingers have touched thegesture surface at locations 875-879. In FIG. 53B, these fingers eachmove to new locations 880-884, respectively. Assume that the gesturecompletes at this point. The arrows 885-889 indicate the backwarddirections for each finger. If the host does not provide fingeridentities for the fingers, the gesture system does so by firstidentifying the thumb by any of the various methods. These methodsindicate that the finger at location 884 is the thumb. The fingers atlocations 880-883 are then identified according to their relativedistances from the thumb 884. FIG. 53C depicts properties derived fromthe squeeze gesture. It shows the thumb at location 884 as a circleddot. The dashed arrows 890-893 point in the reverse direction of thebackward direction arrows 885-888, indicating the finger forwarddirections. The dotted line 180 indicates the horizontal, and thecircled asterisk 894 is a hand location, which in this case is anaverage location for the fingers at locations 880-883. θ is the generalextension angle, here computed as the nearest average of the fingerforward angles. A heavy arrow 895 originating at the hand location 894has direction angle θ and represents the general direction of fingerextension. The second arrow 896 originating from the hand location 894represents the thumb vector. FIG. 53D depicts the result of virtuallyrotating the plane of the gesture surface through an angle of −θ(negative θ). The half-plane 897 above the horizontal line 180 indicatespositive angles and hence the right hand, while the half-plane 898 belowthe horizontal line 180 indicates negative angles and hence the lefthand. The thumb vector 896 falls within the half-plane of negativeangles 898, indicating that the squeeze gesture was performed with theleft hand.

Although the squeeze gesture provides a hand calibration with enoughinformation to produce a finger map, the squeeze gesture is also usefulon touch devices that directly provide finger identities. It may be thatthe touch device is only able to determine finger identities and notalso able to determine which hand is gesturing or the orientation of thehand. The squeeze gesture can still provides this information. Thegeneral value of the squeeze gesture is actually in its ability toprovide the information necessary to distinguish the left hand from theright hand.

7.11.4.3 Squeeze Gesture Properties

The squeeze gesture provides many properties that the gesture system canuse to establish a hand calibration and interpret finger-mappedgestures. These properties derive from one or more paths that thefingers travel while performing the squeeze gesture. Properties of thesqueeze gesture include the following, although they are not limited tothe following:

-   -   1) The start or end locations of the fingers.    -   2) An average location for a set of fingers.    -   3) Relative locations of a set of fingers.    -   4) The path along which a finger travels.    -   5) The point at the center of a path along which a finger        travels.    -   6) The distance along which a finger travels.    -   7) The direction in which a finger travels.    -   8) The average direction in which fingers travel.    -   9) Relative directions in which fingers travel.    -   10) The direction in which a finger travels relative to a        location of another finger.    -   11) The order in which the fingers touch the gesture surface.    -   12) The number of fingers in the gesture.    -   13) The length of time that the fingers touch the gesture        surface.

Properties of a squeeze gesture may also derive from these properties.One of the most useful properties employs information about the thumb.The presence of the thumb establishes an asymmetry in the gesture thatthe gesture system can use to identify the hand, its fingers, or itsorientation. This asymmetry occurs in the positioning of the thumbrelative to the rest of the hand. This property is aspatial-relationship called the “hand-relative positioning of thethumb.” Identifying the index finger as the finger nearest to a locationof the thumb is an example of employing the hand-relative positioning ofthe thumb. Another example is determining the identity of the handaccording to the side of a general extension line on which a thumblocation falls, as previously described. The direction in which thethumb moves can also employ the hand-relative positioning of a thumb ifidentifying the thumb entails comparing locations for the thumb withlocations for other fingers or if the direction of the thumb is relativeto directions or locations for other fingers. As these examplesdemonstrate, the hand-relative positioning of the thumb can be expressedas relative positions of locations for the thumb and non-thumb fingers,as differences between directions of movement for thumb and non-thumbfingers, and as comparisons of directions and relative positionsdetermined from the paths of both thumb and non-thumb fingers.

7.11.4.4 Hand Distinction

A “hand distinction” is a collection of information that suffices todistinguish the left hand from the right hand. The squeeze gestureprovides a hand distinction by virtue of providing all of theinformation that the calibrating system needs to identify the hand. Forexample, the hand-relative positioning of the thumb and the direction offinger extension together determine the hand. It is possible for a handdistinction to only require some information from the squeeze gesture,acquiring the remainder from the host, even if the squeeze gestureitself includes all of the information. In this latter case, the squeezegesture may still be critical for providing missing information.

Consider a touch device that designates a top, bottom, left and rightside of the gesture surface. Modern smart phones and touch tablets dothis. These devices typically detect their orientations and assign topand bottom, left and right accordingly. The application, including thegesture system, then employs this designated orientation. When abidingby this orientation, the squeeze gesture does not arbitrarily establishdirections of finger extension, nor does it arbitrarily establish leftand right sides for the user. Instead, directions of finger extensionare towards the top of the touch device, and the left and right sides ofthe device also correspond to the left and right sides of the user. Thepre-designated general direction of finger extension establishes theorientation of the hand, but it does not distinguish the hands. For agiven hand orientation, the hand-relative positioning of the thumbsuffices to determine whether the hand is the left hand or the righthand.

If the host both provides finger identities and designates the deviceorientation, there may be no need for the user to perform a squeezegesture, unless the gesture system assigns different user requests tothe different hands. In this case, any means for indicating the handidentity suffices, and the squeeze gesture is certainly one means fordoing that. On a system for which the host provides finger identities,once the calibrating system has identified a squeeze gesture, thecalibrating system can use the hand-relative positioning of the thumb toidentify the hand. It is sufficient to use the direction in which thethumb moves during the gesture to identify the hand, because in thisscenario the direction of the thumb with respect to the designateddevice orientation differs by hand.

Now consider a touch device that does not designate a top, bottom, left,and right. Assume that it also does not provide finger identities. Inthis scenario, the host provides no information for distinguishing thehands or determining the hand orientations. The user provides all ofthis information, and the squeeze gesture is a simple, convenient methodfor doing so. A user can gesture on the device with either hand in anyorientation. If the gesture surface is large enough, multiple users canapproach the device from any direction to gesture from where they stand.A community touchscreen table would be an example of such a device. Fromany position around the table, a user need only touch the table with asqueeze gesture to establish a hand calibration at the user's locationand orientation, with the particular hand that the user chose. In thisscenario, the gesture system generates a finger map from properties ofthe squeeze gesture. As previously explained, these properties are alsosufficient to distinguish the left and right hands.

Finally, suppose that the host does not designate an orientation butthat it does provide finger identities. A user can approach the deviceto gesture from any direction. The host knows the finger identities ofthe gesturing fingers, but it may not be able to determine which hand isgesturing when user requests are different for the left and right hands.Even if the host is able to determine which hand is gesturing, it maynot know the orientation of the hand, in which case it would be unableto determine whether any finger is moving forward or backward. Considerthat five fingers in an approximate line on the gesture surface canrepresent either the left hand with fingers directed in one direction orthe right hand with fingers directed in the opposite direction. Agesture system that determines gesture direction would need to know thehand's orientation. As before, the squeeze gesture can provide thisinformation. Under previous scenarios, the calibrating system used thehand-relative positioning of the thumb to distinguish the left and righthand, but the calibrating system need not do so in the present scenario.The host has already provided a series of finger identities, so thecalibrating system can identify the hand by a direction of fingerextension that the squeeze gesture indicates.

In some scenarios, the hand-relative positioning of the thumb issufficient to complete a hand distinction. In other scenarios,identifying the forward or backward direction of finger movement issufficient to complete a hand distinction because this restricts thepossible orientations of the hand. Some scenarios require both of theseproperties to provide a hand distinction. The properties required for ahand distinction depend on the properties that the host alreadyprovides.

7.11.4.5 Reverse Squeeze

It is possible to input a hand calibration by performing the squeezegesture in reverse. This alternative calibration gesture is called a“reverse squeeze.” The fingers of a reverse squeeze start at the endlocations of the analogous squeeze and end at the start locations of theanalogous squeeze. FIG. 52B shows a hand performing a reverse squeeze,with arrows (e.g. arrows 867-868) exemplifying the directions in whichthe fingers may move. The hand would likely need to start the gesturewith fingers more flexed than shown in the illustration. As with thestandard squeeze gesture, the direction angles of adjacent non-thumbfingers are roughly parallel. These are angles θ_(A2), θ_(E2), θ_(C2),and θ_(D2) in the figure. The direction angle of the thumb θ_(E2) canreasonably be expected to differ from the direction angles of the otherfingers by at least π/3 radians by class equivalence. The thumb movesaway from the hand in the reverse squeeze, indicating the position ofthe thumb relative to the hand and thus a hand-relative positioning ofthe thumb.

The process for determining a hand calibration from a reverse squeeze isthe same as for a standard squeeze gesture, except that all referencesto start locations are exchanged with references to end locations.However, the start locations of a reverse squeeze should serve as thefinger rest locations, unless the paths of the fingers in the reversesqueeze represent channel line segments, in which case the midpoint ofthese line segments could serve as the rest locations. Although thereverse squeeze does not end with the fingers in their rest locations,the reverse squeeze does have an advantage over the standard squeezegesture: when using the start locations as the rest locations, thereverse squeeze can be recognized the instant that all of the necessaryfingers have exceeded a gesturing threshold, without waiting for gesturecompletion. The standard squeeze gesture does not have reliable restlocations until the gesture is complete. It may still be beneficial towait for the fingers to finish gesturing in order to more accurateassign forward directions, at least with calibrating systems that needdirections of finger extension. The standard squeeze and the reversesqueeze are variants of a general class of gesture called a “generalsqueeze gesture.”

7.11.4.6 General Squeeze Procedure

The procedure for detecting a standard squeeze is generally the same asthat for detecting a reverse squeeze. FIG. 51 illustrates this procedurein the context of inputting a finger-mapped gesture. In step 860, thecalibrating system first detects the required number of fingers movingto perform the gesture. If this number exceeds the number of fingersrequired for any finger-mapped gesture, there may not be a need to checkthe fingers to determine whether they are moving. Ideally, thecalibrating system would look for at least a certain number of fingersexceeding a gesturing threshold. In step 861, the calibrating systemdetermines that one of the fingers has characteristics on the basis ofwhich it will be treated as a thumb, such as by making the determinationusing any of the previously described techniques. The attempt toidentify a thumb may fail if the calibrating system enforces limitationson the characteristics of squeeze gestures. Upon identifying the thumb,the calibrating system may check for any other constraints that asqueeze gesture must satisfy and only proceed with recognizing thegesture if the constraints are met.

Once the gesture is determined to be a squeeze gesture, in step 862, thecalibrating system may establish or redefine an input area. It may bethat the input area pre-existed the squeeze gesture and is only slightlyreconfigured by the squeeze gesture. For example, if the host providesfinger identities and a left-right orientation for the gesture surface,the only additional information that the squeeze gesture provides is thehand identity. It may also be that the squeeze gesture entirely definesthe input area, such as by establishing the locations and orientationsof geometric elements for input regions, along with indicating forwarddirections for each finger and perhaps also the lengths of the regionsaccording to the lengths that the fingers gestured during the squeeze.Finally, it may be that the squeeze gesture provides an intermediatespecification of the input area, such as by indicating the orientationof a finger map or the directions of finger extension. The dependencythat the input area has on the squeeze gesture can vary greatly bygesture system.

Upon establishing the input area, the gesture system is able to receivegestures via steps 121 and 122. As usual, in step 863, each user requestis a function of the selected identity set and the gesture parameters.However, as described in step 863, the user request is also a functionof at least one property of the general squeeze gesture. Morespecifically, the user request is a property derived from one or morepaths of the fingers that participated in the gesture. If the squeezegesture provided a finger map or indicated the hand identity, theselected identity set will be a function of the properties of thesqueeze gesture that provided this information. It may be that the hostprovides finger identities, in which case the selected identity set neednot be a function of properties of the squeeze gesture. Even so, thesqueeze gesture may provide forward directions for the fingers or thehand identity, and if any of the gesture parameters depend on theseproperties, the gesture parameters will themselves be dependent onproperties of the squeeze gesture. There are many ways for the userrequest to depend on properties of the squeeze gesture, whether throughdependence on the input area defined according to a hand calibrationbased on the squeeze gesture or through dependence on properties of thesqueeze gesture that gesture parameters employ.

7.11.5 Repositioning Gestures

When a calibration gesture gets to a state where a hand calibration isassociated with the gesture but the fingers remain touching the gesturesurface, the gesture can transform into a repositioning gesture. A“repositioning gesture” is a calibration gesture in which a userpositions, orients, and scales an existing hand calibration by movingfingers that are touching the gesture surface to new locations on thegesture surface. The touching fingers have finger identities andlocations by virtue of having performed a calibration gesture. Movingthese fingers moves the rest locations for the finger identities, thusmoving the hand calibration. Rotating the fingers relative to each otherreorients the rest locations and thus rotates the hand calibration.Bringing the fingers closer together or farther apart spaces the restlocations and thus scales the hand calibration. In calibrating systemsthat support repositioning, the user can fine-tune and relocate the handcalibration after performing a calibration gesture and before liftingthe fingers.

For example, assuming that the calibrating system supports repositioningafter drumming, upon performing a drumming gesture to place a handcalibration, a user can leave the fingers on the gesture surface andmove them around to adjust the hand calibration. Each finger of thedrumming gesture has a finger identity by virtue of having performing adrumming gesture, and each has a location by virtue of touching thegesture surface. The user may even lift fingers and still reposition thehand calibration, provided that at least one finger is still touching.If only one finger is touching the calibrating system can onlyreposition the hand calibration without rotating or scaling it, but evenif only two fingers are touching, their relative positions allow forrotating and scaling as well, because their finger identities are known.The calibrating system may even allow fingers to accidentally lift andtouch again by deducing the newly touched finger identities from thepreviously touching finger identities. The sample implementation of theinvention included with this specification describes a “calibrationdetector” that supports accidental lifting during repositioning,allowing the user to touch again to continue repositioning.

To reposition a hand calibration after a squeeze gesture, the user wouldhave to signal completion of the squeeze gesture before dynamicallyrelocating the fingers. For example, the user would have to first holdthe fingers for a timeout period or lift and touch a finger again.Dynamic repositioning behavior is thereafter the same as withrepositioning the drumming gesture.

It is also possible to perform a repositioning gesture on an existingfinger map without first performing a gesture that inputs a new handcalibration. To perform this repositioning gesture, the user places onefinger in each input region of the finger map, placing them in anyorder, thus assigning an identity and a location to each finger. Theuser can then move the fingers around to reposition the hand calibrationexactly as if the gesture had first specified the hand calibration, withthe new finger locations becoming the new rest locations. However, thisrepositioning gesture potentially conflicts with drumming and may bemost useful on systems that employ squeeze gestures rather than drumminggestures.

7.11.6 The Tickling Gesture

Like the repositioning gesture, the “tickling” gesture is a calibrationgesture that must be performed on an existing finger map. It requires anexisting finger map because it does not itself associate fingers withfinger identities. The tickling gesture is performed by each fingerindividually, but multiple fingers can perform it simultaneously. Afinger that is performing the tickling gesture is said to be “tickling.”A finger tickles by touching an input region and moving rapidly backwardand forward in roughly opposite directions—an action that actually lookslike tickling. The tickling gesture establishes a channel line segmentfor the region tickled, also defining a rest location and a forwarddirection for repositioning and shaping the region. The gesture systemcan periodically update the finger map as the user tickles. Becausetickling only acts on an existing finger map, it is most usefullyimplemented by the gesture system.

Recall that the gesture surface abstracts finger positions to a commoncoordinate system, enabling finger identities to be genericallydetermined via finger maps. If the device directly reports fingeridentities, the gesture system need not actually implement a finger mapbecause the finger identities are already known. Tickling is clearlyalso possible in this case as well. While this scenario does not usetickling to further refine a finger map, tickling is still useful inthis scenario for defining forward directions for each finger identity.

A finger roughly traces line segments as it moves backward and forwardto tickle. Each line segment is a vector in the sense that it has astart location and an end location. Each vector also has at least aminimum expected length. Vectors can be detected by monitoring eachfinger's path for reversals in direction. Monitoring for extent vectorsaccomplishes this. A finger is considered to start tickling if it tracesat least a number of extent vectors equal to a “minimum tickling vectorcount” in at most an amount of time equal to the “maximumstart-of-tickling time.” Once a finger starts tickling, the gesturesystem can become more lenient and allow the user to slow the rate oftickling without the gesture system ceasing to detect tickling. Atickling gesture ends when the finger lifts or when the finger tickles anumber of vectors equal to the minimum tickling vector count or fewer inan amount of time greater than the minimum end-of-tickling time.

As a finger with a particular finger identity tickles, the gesturesystem collects the tickling vectors that the finger inputs. The gesturesystem never keeps more than a maximum number of the most recenttickling vectors for any given finger, thus maintaining a window ofrecent tickling vectors. This number is the “tickling vector window.”Periodically during collection, the gesture system averages together thevectors currently in the finger's tickling vector window to produce anaverage vector. In a gesture system with designated top and bottomsides, a collection of vectors can be averaged by averaging the topvector termini together to produce an average top location and thebottom vector termini together to produce an average bottom location.The “average vector” is the vector that starts at the average bottomlocation and ends at the average top location. This vector has two endpoints and is also a line segment. In gesture systems that do not havedesignated top and bottom sides, top and bottom can be determinedrelative to the existing hand calibration, with the average knownforward direction of the fingers indicating the “top” direction, such asthe average of forward directions given by the forward angles of theexisting finger calibrations. In this case, the indicated “top”direction can evolve as the hand calibration evolves with tickling.

After averaging the tickling vectors for a finger of a particular fingeridentity, the gesture system creates a new finger calibration for thefinger identity. The average vector becomes the new channel linesegment, and the midpoint of this line segment becomes the new restlocation for the finger. The finger's new forward angle is the angle ofthe channel line coincident with the line segment directed towards thetop of the device. The new finger calibration replaces the previousfinger calibration for the finger identity, changing the handcalibration and consequently the finger map that derives from it. If thegesture system supports different gesture level threshold distances fordifferent fingers, the threshold distance for each gesture level of afinger can be set as a constant multiple of the length of the finger'saverage tickling vector. It is also an option for the gesture system toaverage the lengths of the average tickling vectors and set thethreshold distances from this average. Alternatively, the gesture levelscan be a function of a determined hand breadth.

The gesture system periodically updates the hand calibration fortickling vectors at a recurring timeout interval that runs while theuser is tickling. The gesture system updates both the fingercalibrations and the finger map at each timeout interval. After updatingthe finger map, subsequent newly tickling fingers should derive theirfinger identities from the latest finger map. This way the finger mapgradually evolves as the user tickles, and the user receives feedbackduring the process, enabling better finger calibrations.

The gesture system continues to collect vectors for a tickling fingerfor as long as the finger tickles. The finger calibration only reflectsthe average of the tickling vectors in the finger's recent ticklingvector window. This allows the user to keep tickling until finding acomfortable finger map, without forcing the user to have to live with afinger map constrained by all past tickling gestures.

FIG. 43C, described previously, illustrates vectors potentially input bytickling fingers. It depicts an average location (e.g. location 737) forthe top locations of the line segments and an average location (e.g.location 738) for the bottom locations of the line segments. The dashedlines 713-715 through these average locations serve as channel linesthat the line segments specify. Rest locations are not shown and are notneeded in this particular finger map because the channel lines are usedto fully partition the containing surface allocation into input regions701-703.

FIGS. 44A-44D show some examples of modifying a hand calibration and itsassociated finger map through tickling. Each figure shows an input areathat covers the entire surface allocation, partitioning it into inputregions 750-752. Dashed lines 753-759 depict channel lines for the inputregions 750-752, taken from the finger calibrations in the underlyinghand calibration. The connected dots (e.g. dots 748-749) representlocations on the paths of fingers as they input tickling vectors (e.g.vectors 768-770). Although each series of tickling vectors is shownentirely contained within a region, the vectors can cross regionboundaries, because the gesture system associates each collection oftickling vectors with the finger identity for the region in which thefinger originated. Circled dots (e.g. dot 749) represent the locationson the vectors that are closer to an indicated top side of the gesturesurface. Each circled asterisk 760-763 indicates the average toplocation of series of tickling vectors, and each asterisk not circled764-767 indicates the average bottom location. The transition from FIG.44A to 44B shows how tickling performed for one finger identity canremake the hand calibration and the finger map. The new channel line 756for region 752 passes through the average top location 760 and theaverage bottom location 764 determined for the tickling vectors, whichis also through an average tickling vector. In a channel-based fingermap, this reshapes regions 751 and 752. FIG. 44C shows tickling on allthree regions 750-752 of this newly formed finger map. When the gesturesystem updates the hand calibration and the finger map again, the resultis the finger map with new channel lines 757-759 shown in FIG. 44D.

Note that the tickling gesture is not generally compatible withmulti-reversal gestures. To support both tickling and multi-reversalgestures, the gesture system can provide a mode in which it does notrecognize multi-reversal gestures, such as by using an escape hold. Forexample, double-touching the thumb could escape the user from fingermapping so that the fingers can perform a tickling gesture that thecalibrating system would unambiguously recognize.

7.11.7 Calibration Reservation

In order to allow users to dynamically place finger maps on the gesturesurface, the calibrating system should remove outdated finger maps. Oneway to manage finger maps is to allow only one finger map at a time andto remove a previous finger map when a new finger map is provided, suchas each time the user performs a new calibration gesture. However, acalibrating system may support a person gesturing simultaneously frommultiple locations, and it may support multiple people gesturing frommultiple locations, whether simultaneously or not. To support fingermaps in multiple locations, it should be possible for users tointentionally add and remove finger maps from the gesture surface. Thecalibrating system can leave finger maps around and require a user toexplicitly perform an action to remove a finger map, but the calibratingsystem can also be intelligent about removing finger maps automatically.For example, the calibrating system can remove a finger map after atimeout period during which no gestures are received, or it could leavea finger map in place until another is input elsewhere for the sameuser.

An effective technique for controlling the lifetime of a finger map isto leave it in place only for as long as the user keeps at least onefinger in contact with it or with the input area. Once the fingers havelifted, after a short timeout period, the calibrating system removes thefinger map. Gesturing is sufficient to continue to reserve a finger map,but the user can also reserve the finger map during pauses in gesturingsimply by letting fingers rest on the finger map or input area. Thesefingers would not register as gestures unless they exceed a gesturingthreshold. Some calibration gestures allow the thumb to rest on theinput area during the gesture, and under this approach it is an optionfor the user to continue reserving the finger map after performing acalibration gesture simply by leaving the thumb in place. If there isanother finger that the gesture system does not normally use, thatfinger can also reserve the finger map while it touches.

Calibration reservation techniques also apply to devices that providefinger identities without requiring the gesture system to implementfinger maps. At minimum, when a calibration gesture identifies the leftor right hand, the user can perform the calibration gesture simply toinform the calibrating system of the hand now in use. The user couldalso perform a calibration gesture to locate a hand on the gesturesurface, such as to establish a localized input area for the hand. Thistechnique is helpful for enabling a calibrating system to distinguishthe finger-mapped gesturing of multiple people on a single gesturesurface. For example, the user could perform a drumming gesture or asqueeze gesture to position an input area. Although rest locations,orientation lines, and forward directions are primarily used forestablishing finger maps, they can also be used for sizing and orientingan input area according to its associated calibration gesture. Once aninput area that does not have a finger map has been established, it canbe reserved using the techniques described for reserving input areaswith finger maps.

7.12 Home Position Drift

As the user inputs finger-mapped gestures over a period of time, thehand is likely to gradually drift across the touch device. When the handdrifts too far from the home position, the gesture system can startproducing unexpected results, forcing the user either to move the handback to the proper position or to perform a calibration gesture at thenew position. It is an alternative for the gesture system to insteaddetect the hand drifting and dynamically move the finger map to staywith the hand. This process is called “dynamic recalibration.” Thegesture system can also help prevent excessive drift by warning the userwith “guard borders.” This section describes these two approaches tohandling drift. The approaches can be used in combination.

7.12.1 Dynamic Recalibration

As a user's hand drifts, the rest locations of the fingers, the forwarddirections of the fingers, and the hand's orientation may change. Thegesture system can optionally detect these changes and automaticallyrevise the hand calibration and finger map to accommodate them. Thisprocess is called “dynamic recalibration.” Dynamic recalibration updatesthe hand calibration based on the paths that the fingers travel as theyinput finger-mapped gestures.

7.12.1.1 The Recalibration Process

Dynamic recalibration can update finger rest location or finger forwarddirections or both. Updating either is sufficient for updating thehand's orientation. The process is best performed by monitoring fingersfor longitudinal movement, although there is no need to do so if theuser or the device restricts fingers to just longitudinal movement. Theprocess is as follows:

-   -   1) The gesture system records descriptions of at least the        finger-mapped gestures that the user performs longitudinally. If        dynamic recalibration is only updating rest locations, the        gesture system need only record the initial touch location of        each finger of the selecting finger set. If the gesture system        is to update the finger forward directions, such as finger        forward angles, it can collect the first extent vector that each        finger inputs during a gesture. The second extent vector of a        gesture is the backbrush vector and is not necessary for the        recalibration. Each description of a gesture includes a        description for each finger of the gesture, associating that        description with the finger identity assigned to the finger.    -   2) The gesture system keeps a list of descriptions of the most        recently input finger-mapped gestures. The list can be limited        to at most a maximum number of gesture descriptions, or it can        be limited to just the gesture descriptions produced within a        certain amount of time in the past. Descriptions for gestures        that exceed the established limit can be discarded.    -   3) The gesture system periodically updates the hand calibration        and the finger map according to the list of recent gesture        descriptions. The gesture system updates the finger calibration        of each finger identity for which at least a minimum update        count of finger descriptions have been collected. The new finger        calibration is derived from the finger descriptions and may also        be derived in part from the previous finger calibration. The        recurring period over which the gesture system performs this        update is either a certain interval of time or every so many        finger-mapped gestures. The gesture system may also monitor an        interval separately for each finger, in which case it is an        option to update a finger calibration after every certain number        of finger descriptions collected for the finger identity.        Depending on the number of gesture descriptions collected and        the update interval, the gesture system may use aging finger        descriptions in multiple successive recalibrations.

7.12.1.2 Updating Finger Calibrations

There are several approaches to updating a finger calibration from thefinger descriptions collected for a particular finger identity. Some ofthese approaches follow:

-   -   1) The initial touch locations of the fingers are averaged        together to produce a new rest location for the finger. This        approach may produce an inaccurate result if the majority of        fingers go in the same direction. For example, if the user only        gestures backward by starting at the top of the input region,        the rest location may be improperly moved to the top of the        input region. If fingers had also gestured forward starting from        the bottom of the input region, the new rest location would more        accurately be located midway between where the forward and        backward gestures began.    -   2) The initial touch locations are averaged together separately        for each discrete direction that the fingers travelled. This        yields one average location for each discrete direction        represented among the finger descriptions. The average locations        for which both opposing directions are represented are then        averaged together, and this final average location is the new        rest location. For example, if the series of finger descriptions        includes fingers that gestured forward, backward, and left, but        never right, the initial touch locations for the forward and        backward finger calibrations are averaged separately, and then        these separate averages are averaged together. The average for        the left direction is not included because there are no gestures        for the right direction.

3) The initial touch locations are averaged together separately for eachdiscrete direction, and opposing average locations are then averagedtogether, as in the previous approach, but with an additionalconstraint. The additional constraint is that each discrete direction tobe included in the average must be represented by a minimum number offinger descriptions. For example, if the minimum number of fingerdescriptions for a direction is 10, and if there are 12 forwardgestures, 9 backward gestures, 10 left gestures, and 2 right gestures,no initial touch locations are averaged together, and no new fingercalibration is computed. This is because no direction and its oppositeboth achieved the minimum number of finger descriptions required. Inthis case, the gesture system keeps collecting gesture descriptions fortesting at the next update interval. This approach ensures that new restlocations are only computed when sufficient data is available.

-   -   4) The gesture system averages the first extent vectors for        fingers travelling in the forward and backward directions and        uses this average vector as the channel line segment of the        finger calibration. Extent vectors are averaged together by        averaging the top termini together to produce a new average top        terminus and by averaging the bottom termini together to produce        a new average bottom terminus. “Top” and “bottom” termini are        determined either relative to a top and bottom of the device,        perhaps as assigned by the host, or relative to the current hand        calibration, as described previously. Since the gesture system        may also be collecting gestures having directions other than        forward and backward, it is possible that there are few or no        forward and backward finger calibrations to use. The gesture        system may opt to increase accuracy by only computing a new        finger calibration when a minimum number of forward and backward        gestures are available.    -   5) The previous approach of simply averaging forward and        backward finger descriptions may prove inaccurate when the        majority of gestures are all in one direction. The channel line        segments of the previous approach can be made more accurate by        requiring a minimum number of forward gestures and a minimum        number of backward gestures.    -   6) Perhaps the most accurate finger calibration results from a        hybrid of an initial touch location approach and an average        extent vector approach. The average rest location for a        particular finger identity is determined according to one of the        above approaches for determining the new rest location from        initial touch locations. The new forward direction for the        finger is computed as an average of the angles of the forward        and backward extent vectors, including each backward extent        vector angle in the average only after adding it to it to direct        it in its implied forward direction. This approach may also be        best when only computing channel lines, where rest locations are        not needed. The gesture system may require a minimum number of        forward gestures and a minimum number of backward gestures to        ensure an accurate forward direction calculation.    -   7) It is possible that a hybrid approach that separately        computes rest locations and average extent vectors ends up        determining a rest location that does not coincide with the        channel line through the average extent vector. In this case, it        may be useful to approximate the new average rest location as        the location on the channel line nearest to the computed average        rest location. Determining the nearest point on a line to        another point is standard geometry. This approach gives        precedence to channel lines over rest locations.

7.12.1.3 Optional Recalibration Features

Dynamic recalibration can benefit from any of the following additionalfeatures, each of which is optional:

-   -   1) The gesture system may restrict itself to collecting only        gesture descriptions for gestures with fingers that move        longitudinally. This prevents lateral gestures from affecting        the finger map, which is primarily intended to accommodate        longitudinal movement.    -   2) Users may take breaks during gesture input, and during these        breaks the hands are likely to reposition a little. When the        hands reposition, the gesture history collected prior to the        break likely does not apply to the new hand position and so        should not be included in any further dynamic recalibration. The        gesture system can preclude old gestures by implementing a        staleness period. Gesture descriptions for gestures older than        the staleness period are discarded from the list of gesture        descriptions used for dynamically computing new hand        calibrations.    -   3) Because a hand may be in a slightly new position after the        user takes a break, it may be helpful to reduce the number of        finger descriptions required to produce the first update to the        hand calibration, in order to more quickly register the hand's        new position. After the first update, the update interval        returns to the normal, longer period.    -   4) A user may not actually wish for the hand to move much during        the gesture and may prefer that the gesture system help enforce        the hand's location on the gesture surface. The gesture system        can accommodate this user while still providing some dynamic        recalibration by only adjusting the hand calibration slightly        towards the newly computed rest location or forward direction.        Upon computing a new rest location or forward direction for a        finger, the gesture system also has the finger's previous rest        location or forward direction. Instead of changing the rest        location to the newly computed value, the gesture system can        change it to a location between the newly computed rest location        and the previous rest location, such as halfway between them.        Likewise, the gesture system can change the forward direction to        a direction between the newly computed forward direction and the        previous forward direction, such as to the nearest average of        the finger's forward angles.    -   5) The user's hand may drift so much that the user is improperly        entering gestures. Because these gestures do not represent the        user's intent, their descriptions should not be used to generate        new hand calibrations. The gesture system can avoid computing        new hand calibrations for improper gestures by recognizing user        requests to undo previous gestures and dropping the gesture        descriptions for those gestures. The user request to undo a        gesture need not actually be input as a finger-mapped gesture;        the gesture system recognizes the user request however it is        input. The gesture system either designates a certain number of        the most recent gesture descriptions as tentative or designates        all gestures younger than a certain age as tentative. When it        periodically calculates the new hand calibration, it only uses        gesture descriptions it has collected older than these tentative        descriptions. For example, the gesture system may keep the six        most recent gestures as tentative. This allows up to six        sequential input mistakes that will not affect the dynamically        computed hand calibration, provided that the user undoes the        gestures before entering a seventh gesture. When the gestures        represent inserted characters, a gesture to delete the preceding        character would be an undo gesture, as would a gesture to delete        the previous word.

7.12.2 Guard Borders

The input area may be located on the touch device near portions of thedevice that the user should be careful not to touch while inputtingfinger-mapped gestures. For example, the input area might appear on atouch-sensitive surface immediately below user interface elements, andaccidentally touching those user interface elements could haveunintentional consequences. As another example, consider the “GoogleNow” feature of phones running the Android operating system. The “GoogleNow” screen can be opened by dragging the finger from below the keyboardarea of the screen onto the keyboard area. If the keyboard area insteadsupports finger-mapped gestures, the user may accidentally brush afinger up from below the keyboard area and open up the “Google Now”feature while intending to input a finger-mapped gesture. Theseaccidents become more likely to happen as the user's hand drifts whilegesturing. The gesture system can implement “guard borders” to helpprevent these potential problems.

A “guard border” is a region at the periphery of the input area on thesides of the input area where sensitive user interface is located. Whenthe user touches the guard border, the user is alerted to this fact. Thealert can be a sound or a flash of color on a screen or some sort oftactile feedback, such as vibration or haptic feedback. The alertinforms the user to adjust gesturing to prevent fingers fromaccidentally touching past the guard border. The guard border can beimplemented so that an alert is only issued when a finger's initialtouch location occurs within the border, or it can be implemented sothat an alert is issued whenever a finger ventures into the border.

If the guard border is strictly between the input area and the sensitiveuser interface, either the host or the gesture system can implement it.The guard border may also be placed on the input area proper, at itsperiphery, in which case the gesture system implements the guard border.In this case, when the user touches the guard border, the gesture systemboth alerts the user and processes the gesture normally. This is theleast disruptive approach, as instead of suddenly refusing to recognizegestures as the hand drifts, the gesture system warns the user that thehand is drifting too far while still recognizing gestures. It issufficient to only reports alerts when a finger's initial touch locationoccurs in a guard border, due to the touchscreen convention whereby userinterface elements only receive input events for fingers that originateon them.

The two approaches can also be combined, with a guard border on theinput area, and just outside it, a guard border off the input area. Thegesture system can place the guard border on any side of the input areaor on any combination of sides of the input area. It may also be usefulto make the presence and behavior of the guard border user-configurable.

7.13 Outcome Determination

A user employs a finger-mapped gesture system to input user requests toa host by performing finger-mapped gestures. The user selects the userrequest by parameterizing the gesture. The parameterization is acollection of parameter values that includes a selected combination offinger identities and at least one additional gesture parameter. Whenthe host does not provide finger identities, the finger identitiescorrespond to input regions of the input area, although the gesturesystem may re-associate finger identities to resolve ambiguityconflicts. The gesture system reports the gesture to the host as a userrequest. The user request may be a characterization of the gesture'sparameterization, in which case the host translates the parameterizationinto a behavior. However, the gesture system may instead attempt totranslate the parameterization into an identifier that names a behaviorand report that identifier to the host. In some cases it may be that theparameterization cannot be associated with such an identifier and theattempt fails, in which case the gesture system can either ignore thegesture or report to the host the occurrence of an unrecognized gesture.

A finger-mapped gesture implements at least one path-dependent gestureparameter in addition to implementing a gesture parameter for fingeridentities. To take advantage of the efficiencies available tofinger-mapped gesturing, the path-dependent parameters should be bothlongitudinally selectable and form-dependent. This specification definesgesture parameters for gesture level, gesture direction, shift holds,touch counts, backbrush level, and multi-reversal. Other gestureparameters are possible too, such as whether fingers perform a quickchange of direction before lifting or whether fingers remain touchingfor a period of time before lifting.

Although this specification has defined gesture parameters that areindependent of each other, it is also possible make the determination ofsome parameters a function of the values of other parameters. Anyparameter value that applies a function to information about a gesturein order to determine the parameter value can choose the functionapplied according to the values of one or more other parameters. Forexample, the gesture level and the backbrush level can each apply afunction to translate a distance measure to a particular range ofdistances. The function employed can be selected as a function of theselected identity set, the shift hold in effect, the touch count, or thegesture direction. As another example, the gesture system can choose adirectional partitioning as a function of the selected identity set foruse in determining a gesture's discrete direction.

A system that produces a result from a set of input values ismathematically a function. A gesture parameterization serves as a set ofinput values and a gesture outcome serves as a computed result. Agesture system therefore behaves as a function that maps gestureparameterizations into gesture outcomes. The gesture system selects auser request as a function of the gesture's parameters even as parametervalues become fixed, or determined, during the gesture. Some parametervalues may become fixed as a user performs a gesture, although whichbecome fixed depends in part on the dependencies among parameters.Finger identities should become fixed before determining gestureparameters that vary by finger identity. The touch count should becomefixed once the fingers start gesturing. Depending on the implementation,the gesture's discrete direction may become fixed once the fingers havetravelled a certain distance, and the gesture level may become fixedonce the user begins backbrushing. Each time a parameter value becomesfixed during a gesture, the set of possible user requests that thegesture may yield becomes further restricted. Each step of the way,while a user is performing a gesture, the gesture system may report tothe host the subset of user requests to which the gesture may stillcorrespond. The host can report the remaining possibilities to the user,such as to implement a tutorial program that helps the user learn thegestures.

7.14 Example Gesture Combinations

The groups of FIGS. 54A-54C, 55A-55C, 56A-56C, and 57A-57C eachrepresent a user performing two distinct gestures on a finger map. Ineach group of figures, one of the gestures is a stroke and the other isa two-fingered sweep. FIGS. 54A, 55A, 56A, and 57A each depict an inputarea with the two finger paths 928-935 under consideration for thestroke or sweep of the example. The arrow at one end of each pathindicates the direction in which the finger travels along the path,shown with a distance vector 936-942 extending from the path'smeasurement origin to the finger's lift location. In each group, onedistance vector has length X and the other length Y. FIGS. 54B, 55B,56B, and 57B each depict a concentric-circle “target” illustrating theevaluation of a stroke gesture that uses only one of the finger pathsshown on the input area. FIGS. 54C, 55C, 56C, and 57C each depict atarget illustrating the evaluation of a sweep gesture that uses both ofthe finger paths shown on the input area. In FIGS. 54A, 55A, and 56A,the input area is partitioned into input regions 900-910. Each of theseinput regions is derived from a channel line 915-925. The capital boldletter in each region represents the finger identity of the region.FIGS. 55A and 57A show a barrier 913-914 dividing the input area intotwo facets, representing either a barrier on a single facet or a barrierbetween facets.

The concentric circle “targets” are analogous to those described forFIGS. 35A-35D and 36A-36D, here representing the translation of gesturesinto user requests using an average distance vector. The center of eachtarget represents a measurement origin. The solid vector that extendsfrom the measurement origin, which is one of vectors 930, 936, 940, 941,and 943-946, represents a distance vector for the gesture as a whole.Each concentric circle 947-950 represents a gesture level thresholdlength from the measurement origin. The innermost circle 947 or 949represents a gesturing threshold distance, below which gestures are notrecognized and there is no gesture level. Each dashed crosshair 953-956centered on a target represents an oriented partitioning oriented to areference direction. The reference directions are shown as dashedvectors 925, 926, 951, and 952. The concentric circles 947-950 partitionthe target into concentric rings, and when present, the orientedpartitioning further partitions the concentric rings by discretedirection. Each partitioning of the target contains a single quotedcharacter representing a user request for inputting that character. Theuser request that a target indicates for a gesture is the characterlocated in the partition of the target that contains the ending locationof the gesture's distance vector, which is one of vectors 930, 936, 940,941, and 943-946. The length of the distance vector for the sweepgesture is given by Z and is equal to the average of lengths X and Y.

FIG. 54A depicts four input regions 900-903. Input regions 900-902 havefixed channel lines 915-917. The channel line 918 for region 903 isdetermined from a hand calibration and is not parallel to the otherchannel lines 915-917. Computing the channel-based finger map results inregions 902 and 903 taking forms distinct from those of regions 900 and901. According to the stroke interpretation of this figure, a fingertouches input region 902 and gestures into region 903 yielding aselected distance of X. No other finger gestures in this firstinterpretation, so there is no finger gesturing from region 903.According to the target in FIG. 54B, the stroke gesture is notparameterized by direction, only by fingers selected and gesture level.The target shows the distance vector 936 for finger C having length X.The distance X designates the character ‘E’, so the stroke gestureresults in a user request to input the character ‘E’. Under the sweepinterpretation of this figure, the user is gesturing a two-fingeredstroke with fingers C and D. According to the target in FIG. 54C, theaverage of the distance vectors 936 and 937 is distance vector 943having length Z. Although both the stroke and the sweep landed in thesame ring of the target, the sweep indicates a different character—thecharacter ‘G’—because the combination of finger identities C and D isdifferent from that of just C as used in the stroke.

FIG. 55A also depicts four input regions 904-907, but region 904 is onits own facet. The other three regions 905-907 share a facet. Thisfigure is analogous to FIG. 54A, except that the regions 905-907 areeach based on channel lines 920-922 that are at distinct angles.Computing a channel-based finger map results in each of these regions905-907 having a unique form. Under the stroke interpretation of thisfigure, a single finger gestures in region 906; the finger path depictedin region 907 is assumed not present for this interpretation. The targetin FIG. 55B indicates that the distance vector 930 for finger C selectsthe character ‘F’. Under the sweep interpretation, the fingers depictedin regions 906 and 907 together perform the gesture, indicating fingersC and D. Vector 944 of the sweep target is the average of the twofingers' distance vectors 930 and 938. Because the combination offingers C and D of the sweep is different from the combinationcontaining only finger C as used in the stroke, the indicated gesturelevel selects from a different set of possible user requests. In thiscase, the average distance vector 944 indicates the character ‘G’.

FIGS. 56A-56C and 57A-57C illustrate finger-mapped gestures that arefurther parameterized by gesture direction according to the directionsin which the individual fingers gesture. In these figures, ϕ_(A), ϕ_(B),and ϕ_(C), indicate the angles of the distance vectors for fingeridentities A, B, and C, respectively. Under the sweep interpretation ofthese figures, the gesture angle ϕ is the angle of the average distancevector and can be computed as the nearest average of the angles of thedistance vectors for the fingers that participate in the gesture. Theangle ϕ selects a discrete direction from the oriented partitioning, andthe sweep length Z selects a gesture level. In both the strokeinterpretation and the sweep interpretation, the combination of fingersgesturing, discrete direction, and gesture level together indicate theuser request being selected.

FIG. 56A depicts fingers gesturing on input regions 908-910 that arederived from geometric elements determined according to a hand. Thechannel lines 923-925 depicted may be thought of as those geometricelements, but the channel lines may also have been derived from morecomplex geometric elements. Each circled asterisk (e.g. asterisk 957)represents the average forward location of the underlying complexgeometric element, and each asterisk that is not circled (e.g. asterisk958) represents the average backward location of the underlyinggeometric element. In this example, each channel line is derived fromthe geometric element as the line passing through these two points, suchas channel line 923 being derived from the points 957 and 958. Theangles θ_(A), θ_(B), and θ_(C) are forward angles for the channel lines923, 924, and 925, respectively. Under the stroke interpretation of thisfigure, only finger C gestures, completing with length Y and a directionangle of ϕ_(C). The target in FIG. 56B is for a stroke with finger C.The oriented partitioning 953 of this target is oriented to the forwardangle θ_(C) for region 910. The stroke thus indicates a request inputthe character ‘B’. As shown in FIG. 56C, the sweep interpretation ofthis figure uses an oriented partitioning 954 oriented to the forwarddirection 951, given as reference angle θ, which is computedspecifically for the combination of the gesturing fingers A and C. θequals the nearest average of the angles θ_(A) and θ_(C). Vector 945 isthe average of the distance vectors 939 and 940, having direction angleϕ and length Z. The discrete direction at angle ϕ, the gesture level atlength Z, and the fingers identities A and C together indicate a requestto input the character ‘J’.

FIG. 57A is analogous to FIG. 56A, except that the means for determiningfinger identities is not shown. For example, the touch device coulddirectly provide finger identities. A forward direction is stillassociated with each finger identity, however, and is set via a handcalibration. The vectors 926 and 927 represent forward directions forfingers B and C. A barrier 914 divides the gesture surface into twohalves 911-912. Each half 911-912 could represent its own facet with itsown coordinate system, or the two halves 911-912 could be part of thesame facet by sharing a coordinate system. This example assumes thatthere are at least three possible finger identities given as fingers A,B, and C, even though finger A is not participating in the gesturesshown here. Under the stroke interpretation for the figure, only fingerB gestures in half 911. As shown in FIG. 57B, a directional partitioning955 is oriented to the forward angle θ_(B) associated with finger B. Thedistance vector 941 for finger B provides a sampling angle ϕ_(B) and agesture length X for determining the user request. This particularstroke indicates a request to input the character ‘B’. Under the sweepinterpretation of this figure, both fingers B and C gesture, and thetarget in FIG. 57C represents a determination of the user requestindicated, doing so from a sweep vector 946, which could be computed asthe nearest average of the vectors 941 and 942. However, if the twohalves 911-912 represent distinct facets with distinct coordinatesystems, vector 946 can instead be computed as a vector with length Zand an angle ϕ equal to the nearest average of the angles ϕ_(B) andϕ_(C). Here the directional partitioning 956 is oriented to the forwarddirection 952, whose forward angle θ is the nearest average of theforward angles for fingers B and C, which are angles θ_(B) and θ_(C).FIG. 57C depicts θ happening to be vertical and equal to ϕ. According tothis gesture level and directional partitioning, the sweep indicates arequest to input the character ‘I’.

FIGS. 58A-58C, 59A-59C, and 60A-60B borrow notation from FIGS. 54A-54Dand together depict three series of gestures that include both sweepsand strokes. The first series is given by FIGS. 58A-58C. This seriesdepicts a finger map with four regions 960-963 corresponding to fingeridentities A, B, C, and D, with region 960 on its own facet. The channellines 971-973 are varyingly separated, creating varyingly sized regions961-963. Gestures in this finger map are not parameterized by gesturedirection. In FIG. 58A, a user gestures two fingers along paths 966 and967 to input a sweep. The fingers originate on regions 961 and 962,indicating finger identities B and C. The gesture system evaluates thesweep via an interpolated distance vector 970. According to thediscussion of this notation in previous figures, the gesture indicates arequest to input the character ‘F’. The user lifts the fingers and nextgestures a stroke in FIG. 58B along path 968 starting in region 961,indicating finger B. This stroke indicates a request to input thecharacter T. The user lifts finger B and in FIG. 58C performs anotherstroke along path 969 on facet 960, indicating finger identity A. Thisnew stroke indicates a request to input the character ‘L’.

FIGS. 59A-59C depict another series of gestures. The figures do not showhow finger identities are determined, but the touch device coulddirectly provide the identities. A gesture level and a gesture directionparameterize each gesture. Gesture direction is determined from ahalf-plane partitioning that is oriented according to the identities ofthe gesturing fingers. In FIG. 59A, the user performs a backward gesturelevel 2 stroke along path 975 with finger A, indicating a request toinput the character ‘D’. In FIG. 59B, the user performs a backwardgesture level 1 stroke along path 976 with finger B, indicating arequest to input the character ‘G’. The figure depicts different forwarddirections 978 and 979 for fingers A and B, given as forward anglesθ_(A) and θ_(B), relative to which the respective half-planepartitionings 981 and 982 are oriented. In FIG. 59C, fingers A and Btravel paths identical in form to the paths that they traveled in FIGS.59A and 59B, except that they do so as part of a two-fingered sweepgesture. Although paths 975 and 976 start at different locations in FIG.59C, the figure numbers the paths identically to their numbers in FIGS.59A and 59B to emphasize their identical forms. FIG. 59C combines paths975 and 976 into a single gesture using an interpolated distance vector977. The sweep is here using a half-plane partitioning 983 that isoriented to a forward angle θ that is specific to the sweep. Here θ isthe nearest average of angles θ_(A) and θ_(B). θ happens to be vertical,partitioning the two available discrete directions along the horizontal180. The interpolated distance vector 977 has gesture level 2 andindicates a request to input the character ‘N’.

FIGS. 60A and 60B depict a final series of gestures. As with theprevious series, no regions are depicted and the figures do not show howfinger identities are determined. The gesture system in this examplefurther parameterizes gestures with a gesture direction and supports acorrespondence series of three fingers, identified as A, B, and C. Thegesture system associates a distinct forward angle with each finger.Finger A has a forward angle θ_(A) equal to 7π/12 radians, finger B aforward angle θ_(B) equal to π/2 radians, and finger C a forward angleθ_(C) equal to π/3 radians. In FIG. 60A, the user performs atwo-fingered sweep along paths 985 and 986 with fingers B and C. Theforward angles for fingers B and C—angles θ_(B) and θ_(C), which are notdepicted—are averaged to produce a reference angle θ for orienting adirectional partitioning 991. θ equals 5π/12. The user request can thenbe determined for an interpolated distance vector 988. It is a requestto input the character ‘G’. The user lifts both fingers and then in FIG.60B performs a stroke along path 987 with finger A. The orientedpartitioning 992 is oriented to the forward direction 990 of finger A,which is given as forward angle θ_(A). The gesture indicates a requestto input the character ‘L’.

8 SAMPLE IMPLEMENTATION OF THE INVENTION

The “sample implementation” provided herein summarizes the design of aworking implementation of a finger-mapped gesture system, although theassignment of gesture parameterizations to characters is here shown withimprovements over the assignments used in the working implementation.The design depicts one way to integrate some of the finger-mappedgesturing options. Not all possible options are implemented. Much of thedesign is relegated to detailed state-transition diagrams. See thesection “State-Transition Diagrams” at the end of the sampleimplementation for an explanation of how to read the state-transitiondiagrams.

8.1 Design Characterization

Here are some of the design decisions that characterize the sampleimplementation:

-   -   1) Touch Device—The sample implementation is written for a        single-facet touchscreen device such as a tablet or a        smartphone. The implementation displays the boundaries between        input regions on the touchscreen, even dynamically moving the        boundaries during repositioning. However, it does not need to        display anything, so it would work equally well on a        touch-sensitive surface not capable of display.    -   2) Host—The host is either an application or the touch device        operating system. The host does not detect calibration gestures        but instead relegates that to the gesture system, which is        therefore limited to receiving calibration gestures only on the        provided surface allocation.    -   3) Surface Allocation—The system is designed to work on a        rectangular surface allocation such as the one a touchscreen        device normally designates for a virtual keyboard. An        application may designate the rectangular area instead. One side        of the surface allocation is designated left and the other side        is designated right, one side top and the other side bottom. The        x-coordinate increases from left to right, y-coordinates from        bottom to top (contrary to the device for which it was        implemented, requiring adjustments to the input events).    -   4) Gesture Session—The gesture session persists until the        surface allocation is closed. Finger maps may be dynamically        relocated and re-calibrated on the surface allocation during a        gesturing session. The session monitors for both finger-mapped        gestures and non-finger-mapped gestures using the same        architecture. This is an alternative to having the host        simultaneously monitor the surface allocation for        non-finger-mapped gestures. It also makes for a more flexible        gesturing engine.    -   5) Input Area—The input area partitions the entire surface        allocation into regions.    -   6) Hand Calibration—The system uses a channel-based hand        calibration that always contains four finger calibrations,        representing forward directions as forward angles. It is capable        of representing arbitrary channel lines, as long as all forward        angles are exclusively between 0 and π radians (that is,        directed towards the “top” of the device). However, the system        may deduce finger calibrations from finger calibrations for just        the index and pinky fingers. The hand calibration specifies        whether it is for the left or right hand.    -   7) Finger Map—The finger map has four regions, one for each        non-thumb finger. The boundary between adjacent regions bisects        the area between adjacent channel lines. The outside boundaries        of the end-of-series input regions are the boundaries of the        surface allocation. The finger map can orient in any direction        compatible with the forward angle constraint. The constraint on        forward angles guarantees that the leftmost region always        belongs to either the index or pinky finger, depending on the        hand in use.    -   8) Gesturing Threshold—The gesturing threshold is an initial        slide with a user-configurable gesturing threshold distance and        minimum speed. The minimum speed is given as the maximum amount        of time it takes the finger to travel the gesturing threshold        distance.    -   9) Distance Measure—The starting point of the initial slide is        used as the measurement origin. The distance measure is the        length of a line segment that extends from the measurement        origin to the sampling point.    -   10) Lockstep Gesturing—The only finger-mapped gestures        recognized are strokes and two-fingered sweeps. The gesture        system establishes a discrete direction for the gesture as a        whole once all fingers have travelled a minimum distance. The        gesture must have the same discrete direction at the time the        selecting finger set is determined. The distance vectors of the        two fingers of a sweep cannot have angles that differ by more        than π/2 radians by class equivalence. The gesture level        integers of the individual fingers of a sweep are allowed to        differ by at most 1, ensuring that the distances that the        fingers of a sweep travel are not too different. The gesture        system does not limit the directions of movement available.    -   11) Retraction and Revocation—The sample implementation supports        neither retraction nor revocation, which appear to have limited        value.    -   12) Region Selection—The gesture system calculates region        boundaries from channel lines and compares finger initial touch        locations to these boundaries, associating each finger with its        originating region. If both fingers of a sweep fall within the        same region, the gesture system identifies the region closest to        the midpoint between the two fingers' initial touch locations        and reassigns one of the fingers to another region according to        the midpoint determination process described in the detailed        description of the invention.    -   13) Touch Counts—The gesture system supports multi-tapping for        arbitrary touch counts. A repeat touch is given by either        touching the same input region or by touching within a vicinity        of the original touch, which is in its originating region. The        touch counts of the fingers of a sweep may not differ by more        than one in order for the sweep to be recognized. The touch        count of the sweep gesture is the greater of the two touch        counts.    -   14) Gesture Level—The gesture system supports three gesture        levels, the first of which it designates level 0. Level 0 was        intended to provide a minimum threshold distance for recognizing        gestures, but in practice it proved best to set this distance to        the minimum slide length, making it impossible to select gesture        level 0. The distance that a finger must travel to indicate the        other gesture levels is user-configurable. When no backbrush is        performed, a finger's level selection point is determined for        the locations of the fingers at the time the selecting finger        set is defined. If the finger performs a backbrush, the gesture        level is determined from the lengths of the first extent vectors        of the fingers in the selecting finger set. The gesture level of        a two-fingered sweep is the greater of the gesture levels of the        two fingers.    -   15) Holds—Residual holds are implemented with a        user-configurable timeout. The thumb may rest on the input        region for either the index finger or the middle finger without        affecting finger-mapped gesturing. The thumb may perform an        escape hold by preceding a hold on either of these regions with        a multi-tap touch count of 2 or higher, but this puts the        gesture system into a mode of accepting non-finger-mapped        gestures for the duration of the hold. The touch count selects        the particular set of non-finger-mapped gestures enabled. The        pinky may also perform a shift hold to allow the remaining        fingers to gesture numbers and other characters. Hold locks are        implemented for the thumb anchor, the thumb escape, and the        pinky finger shift.    -   16) Long-Down Feedback—The gesture system detects a finger that        has been placed on the gesture surface for an extended period of        time without gesturing and reports this to the gesture        detectors, which may use this information for specialized        gestures and to provide the user with feedback. The autorepeat        gesture described later is an example of such a specialized        gesture. The gesture system may also use this feature to provide        the user with mid-gesture feedback. If the user touches one or        more fingers for the long-down timeout period, the gesture        system can fix the finger identities and touch counts of the        fingers to identify the set of possible user requests available        to these fingers. The gesture system can report the long-down to        the host, which may then inform the user of the user requests        available to those fingers.    -   17) Gesture Direction—All finger-mapped gestures indicate one of        four discrete directions, FORWARD, BACKWARD, LEFT, and RIGHT,        according to a quarter-plane partitioning that orients        separately to each finger's forward angle. In a two-fingered        sweep, the reference angle is chosen as the average of the        forward angles for the fingers gesturing, and the sampling angle        is compared to the same quarter-plane partitioning oriented to        this reference angle. If sweep does not include a backbrush, the        sampling angle of a sweep is the nearest average of the angles        of the distance vectors for the two fingers at the time the        selecting finger set is determined. If the sweep includes a        backbrush, the sampling angle is the nearest average of the        angles of the fingers' first extent vectors.    -   18) Backbrushing—The gesture system supports two-level        backbrushing, with the distance necessary to travel to select        the second backbrush level user configurable. In a two-fingered        sweep, the backbrush level of the gesture is the greater of the        backbrush levels of the two fingers, and in order for the        gesture to be recognized, the backbrush level integers of the        two fingers cannot differ by more than 1. The tables below that        assign gesture parameterizations to characters include a third        backbrush level to illustrate the potential for extending the        sample implementation to use additional backbrush levels.    -   19) Selecting Finger Set—The selecting finger set is determined        at the time the first gesturing finger lifts. The user request        is also selected at this time, employing the shift holds in        effect at this time to further select the user request. The        fingers of a sweep must be adjacent. This constraint is enforced        to prevent conflicts with spaced-finger non-finger-mapped        gestures that the gesture system allows the user to perform on        top of the input area to close the input area or select        alternate modes of operation. The gesture system does not issue        the user request until all fingers of the selecting finger set        have lifted, and if any finger continues to gesture for more        than a certain amount of time after the first gesturing finger        lifts, the gesture is treated as an error and the user request        is not reported.    -   20) Extent Vectors—The sample implementation monitors for extent        vectors during both finger-mapped gestures and tickling        calibration. The gesture system records the first and the second        extent vectors for use in determining the gesture and backbrush        levels. The calibration gesture logic collects detected extent        vectors over a period of time.    -   21) User Requests—The user requests are identifiers indicating        requests to input characters and to perform other text editing        functions. The gesture system looks up each gesture's        parameterization in a set of structured tables to retrieve the        indicated user request.    -   22) Status Reports—The gesture system reports occurrences of        unrecognized gestures to the host. It also reports occurrences        of hand calibration gestures, shift holds, shift hold releases,        and a variety of non-finger-mapped gestures to the host.    -   23) Home Position Drift—The sample implementation does not        handle home position drift, implementing neither dynamic        recalibration nor guard borders.

8.2 User Request Gestures Supported

The gesture system supports three modes of operation on the surfaceallocation. The modes are cursoring, highlighting, and character entry.Only the character entry mode supports finger-mapped gestures. Thecursoring and highlighting modes receive gestures for moving the cursorand for highlighting text. The design for this sample implementationsupports all of these modes with the same gesture system; the sessionmanager, finger tracker, and session engine implement finger-mapped andnon-finger-mapped gestures alike. The gesture system simply includes agesture detector specialized for each class of gesture. The gesturesystem also monitors for drumming, tickling, and repositioningcalibration gestures. This section describes the supported gestures thatresult in user requests being issued to the host. A gesture detector isalso used to detect calibration gestures, but calibration gesturesupport is described in the “Calibration Detector” section.

8.2.1 Recognized Finger-Mapped Gestures

The implementation translates finger-mapped gestures into text editingrequests. Most of the gestures yield requests to insert characters, andsome are requests to edit previously entered text. The tables in thissection correlate gesture parameterizations with user requests. Thetables are designed specifically for American English and were largelyproduced using principles such as the following:

-   -   1) The tables only use gesture levels 1 and 2. Gesture level 1        is called a “short” gesture, and gesture level 2 is called a        “long” gesture. It appears to be easy to learn two gesture        levels.    -   2) The most frequently entered letters are generally assigned to        the gestures that are the fastest and easiest to enter. In        English, the seven most frequently occurring letters are A, E,        I, O, N, S, and T. They are assigned to short single-finger        strokes not parameterized by multi-tapping or backbrushing.        Gesturing backward is more natural than gesturing forward, so        these are given to the most common letters, the vowels.    -   3) Second in precedence to assigning the most common letters to        easy gestures is grouping gestures alphabetically. The fastest        possible finger map would prioritize assignments by frequency of        entry throughout, but the system would be too intimidating to        learn. By grouping series of gestures alphabetically, the user        need only remember the gesture for the first letter, making the        gestures for the remaining letters of the alphabetical sequence        clear.    -   4) Where possible, letters are assigned to gestures that        approximate the form of the character. For example, a        double-touch long stroke up of the pinky indicates an        exclamation point, which can be drawn by tapping once and        lifting for the dot and touching again and stroking up for the        line, exactly as this gesture is performed. This helps users to        remember the assignments of characters to gestures. Sometimes        the only similarity is the number of pen/finger strokes common        to the character and the gesture.    -   5) Similar characters are generally assigned similar gestures.        For example, the question mark gesture is the gesture for the        exclamation point with a level 1 backbrush added. A dash is a        short stroke to the right, and an em-dash is a long stroke to        the right.    -   6) A number of non-letter symbols are assigned gestures that        differ from the gesture for a letter by only one parameter. The        similarity of the gesture to the letter allows the letter to        serve as a mnemonic for the gesture. For example, single-touch        short strokes forward and backward produce the letters ‘n’ and        ‘a’, while double-touch short strokes forward and backward        produce the symbols ‘&’ and ‘@’, respectively.    -   7) Left and right gesturing fingers produce identical user        requests for the index and middle fingers and identical requests        for the ring and pinky fingers. This allows the user to select        whether to gesture a finger from its rest location region toward        another region or from another region toward its rest location        region. This also helps users learn to gesture with either the        left or the right hand.    -   8) Virtually all assignments are made to fingers moving forward        and backward. Fingers moving forward and backward do not have to        leave the home position, enabling more rapid input. When the        hand leaves home position to move a finger left or right, the        hand must first move back to home position to continue        gesturing, slowing down input and potentially introducing error        by not properly returning.    -   9) No sweeps are assigned left or right movement. This allows        the gesture system to interpret all sweep-like movement left and        right as gestures other than finger-mapped gestures, such as for        opening a virtual keyboard or help screens on the surface        allocation.    -   10) All sweeps are for adjacent fingers only. This allows the        gesture system to provide a special interpretation for gestures        that move forward or backward with fingers spaced apart at least        a minimum distance; these gestures are not finger-mapped. Sweeps        that are restricted to adjacent fingers also appear to take less        dexterity for a user to perform, making them easier to learn and        faster to input. Moreover, two-finger sweeps have an intuitive        left-to-right ordering, which the gesture system takes advantage        of to make it easier for the user to associate characters with        the combinations of fingers involved in the sweeps.    -   11) Some characters are accessible via several gestures. This        allows the user to pick the mnemonic that works best for the        user.    -   12) The number characters are made available via a pinky shift        hold. The pinky shift is analogous to a numeric keypad.    -   13) A letter case mode toggle is available by double-touching        two fingers in input regions that are not adjacent. This        reverses the case of the letters in the tables—lower case mode        becomes upper case mode and upper case mode become lower case        mode. Performing a new calibration gesture resets to lower case        mode.

The tables below express gesture parameterizations using a shorthandnotation. 1×, 2×, and 3× identify touch counts of 1, 2, and 3,respectively. “Short” is gesture level 1, and “long” is gesture level 2.F, B, L, and R identify the discrete direction assigned to the gestureand indicate FORWARD, BACKWARD, LEFT, and RIGHT, respectively. +0, +1,and +2 indicate no backbrushing, backbrush level 1, and backbrush level2, respectively. Each cell shows the indicated user request. Theoccurrence of just a character in a cell indicates a request to insertthat character. The tables are modified from the inventor'simplementation to demonstrate the variety gesture translations possible.Blank cells indicate gestures that do not translate to user requests.The rows for many parameterizations are not shown because they areempty. Glancing through the table conveys an idea of the magnitude ofuser requests that finger-mapped gestured makes possible. Although thetable columns are arrayed left-to-right from index finger to pinkyfinger, each table applies to both the left and right hands. That is,each finger named produces the same user requests regardless of whichhand is gesturing. This facilitates learning to gesture with both theleft and the right hand.

Index Middle Ring Pinky Strokes Finger Finger Finger Finger 1x long F +0 j k l m 1x short F + 0 n s t . (period) 1x short B + 0 a e i o 1x longB + 0 b c d (space) 1x short L + 0 (delete (delete char left) char left)1x short R + 0 (space) (space) - (dash) - (dash) 1x long R + 0 (space)(space) — (em dash) — (em dash) 1x long F + 1 J K L M 1x short F + 1 N ST , (comma) 1x short B + 1 A E I O 1x long B + 1 B C D (backspace) 2xlong F + 0 / { £ ! 2x short F + 0 & [ ~ : 2x short B + 0 @ ) ‘ (s.quote) ° (degree) 2x long B + 0 \ < $ - (dash) 2x short L + 0 (del.(del. word left) word left) 2x short R + 0 (del. (del. _ (underscore) _(underscore) char right) char right) 2x long F + 1 } ? 2x short F + 1 ]{circumflex over ( )} : 2x short B + 1 ) {grave over ( )} (back tick) 2xlong B + 1 > + 2x short B + 2 á é í ó 2x long B + 2 ã ĩ õ 3x short B + 03x long B + 0 — (em dash)

Sweeps Index + Middle Middle + Ring Ring + Pinky 1x long F + 0 x y z 1xshort F + 0 u v w 1x short B + 0 f g h 1x long B + 0 p q r 1x long F + 1X Y Z 1x short F + 1 U V W 1x short B + 1 F G H 1x long B + 1 P Q R 1xlong F + 2 1x short F + 2 {umlaut over ( )} (modifier) {hacek over ( )}(modifier) ~ (modifier) 2x long F + 0 * (tab) 2x short F + 0 _(underscore) | (vertical bar) 2x short B + 0 ° (degree) “ (d. quote) 2xlong B + 0 % = (new line) 2x long F + 1 (back tab) 2x short F + 1{circumflex over ( )} 2x short B + 1 2x long B + 1 # # (line break) 2xlong F + 2 ũ 2x short F + 2 ú

Pinky Shift Strokes Index Finger Middle Finger Ring Finger 1x long F + 07 8 9 1x short F + 0 4 5 6 1x short B + 0 1 2 3 1x long B + 0 0 .(period) (space) 1x short F + 1 , (comma) 1x long B + 1 (backspace) 2xlong F + 0 / 2x long B + 0 - (dash) 2x short B + 1 2x long B + 1 +

Pinky Shift Sweeps Index + Middle Finger Middle + Ring Finger 1x longF + 0 * 1x short F + 0 1x short B + 0 1x long B + 0 % = 1x short F + 1(tab) 1x long B + 1 (new line) 2x long F + 0 (back tab) 2x long B + 0(line break)

FIGS. 61A and 61B serve as cheat sheets for inputting letters andnumbers according to the above tables. FIG. 61A is a finger map whosefour input regions 994-997 are parallel and positioned relative to eachother according to a hand calibration. FIG. 61B is a finger map whosefour input regions 994-997 are equivalent to the input regions of FIG.61A but shaped to accommodate channel lines along which the fingersmove. In both finger maps, adjacent input regions are alternately shadedfrom one side of the finger map to the other. The dotted lines 998-999and dashed lines 1000 do not represent boundaries within the regions994-997, instead providing a mnemonic depiction of the gestureparameterizations available. The dashed line 1000 separates charactersthat are input via forward gestures from those that are input viabackward gestures. The characters shown in the top half 1001 of thefinger map, above the dashed line 1000, are input via forward gestures,and the characters shown in the bottom half 1002 of the finger map,below the dashed line 1000, are input via backward gestures. Inparticular, the characters in the top half 1001 shown above the dottedline 998 are input via long forward gestures and those shown below thedotted line 998 are input via short forward gestures. The characters inthe bottom half 1002 shown above the dotted line 999 are input via shortbackward gestures and those shown below the dotted line 999 are inputvia long backward gestures.

Each input region 994-997 in FIGS. 61A-61B is shown partitioned intofour cells. Each cell represents a stroke input with a particular fingerin a particular direction at a particular length. Were fingers togesture starting on the dashed line 1000, they would move toward thecell that indicates the desired character. The cells are mnemonic onlyand do not functionally correspond to areas on the input region. Mostcells have a top character and a bottom character, although cells in thepinky finger region 997 only have bottom characters. The bottomcharacter is the character that the corresponding gesture normallyinputs. For example, a long stroke forward with a finger anywhere in thering finger region 996 indicates the letter ‘l’, and a long strokebackward with a finger anywhere in the ring finger region 996 indicatesthe letter ‘d’. The character at the top of the cell indicates thecharacter that the corresponding gesture inputs when performed using apinky shift. For example, holding the pinky finger down while gesturingthe middle finger forward a short distance indicates a ‘5’. Thecharacters that span boundaries between the cells of adjacent regionsrepresent sweep gestures input in the given direction for the indicatedlength. For example, a long sweep of the middle and ring fingersbackward indicates the letter ‘q’. These figures do not show the resultsof sweeping one finger a short distance and another finger a longdistance. In these cases, the sample implementation treats both fingersas having performed a long gesture.

8.2.2 Autorepeat Gestures

It is common for a user to need to repeatedly input the same character.A physical keyboard allows the user to do this by continuing to hold akey down. A finger-mapped gesture system should make this an easyoperation too. The sample implementation provides a way to do this, butthe detailed design is not included in this specification.

The obvious solution is to repeatedly report the user request for agesture if the user keeps the fingers touching the gesture surface atlength at the end of the gesture. The sample implementation only detectsthe end of the gesture with the lifting of a finger, so this does notwork in the present context. An alternative implementation could detecta gesture ending when the user's fingers become stationary aftergesturing, but this alternative is not suitable for users who arelearning to gesture because they may often leave their fingers down inhesitation. This alternative is also potentially problematic should thehost or the gesture system support non-finger-mapped gestures that mightbe ambiguous with gestures that are recognized before a finger lifts.

A better solution allows the user to complete the gesture with thelifting of a finger and requires that the user touch the fingers againright afterwards to induce autorepeat behavior. If the user touches thesame fingers that produced the previous gesture and holds them down atlength, the gesture system can detect this as a request to repeat theprevious user request. However, under this approach, users are likely tohesitate their fingers as they learn to use the system, and thishesitation will likely be interpreted a request to autorepeat theprevious user request.

A still more robust solution requires the user to complete the gesture,lift the fingers, and then double-touch or triple-touch the same fingersto induce an autorepeat. For example, the user may perform a long sweepforward with the index and middle fingers to issue an ‘x’ user request,lift both fingers, and then double tap both the index and middlefingers, holding them against the gesture surface. After a timeoutperiod, the gesture system determines that these are the fingers of theselecting finger set of the most recently input gesture and thereforebegins reporting the previous request at regular intervals,autorepeating the ‘x’ while the user holds the two fingers touching. Ifthe user initiates the autorepeat with a triple-touch, the gesturesystem can autorepeat the previous user request at a faster rate,allowing the user to select the rate of the autorepeat.

Autorepeat functionality can be implemented with an autorepeat gesturedetector, which is not depicted in this specification. It has a statemachine having an Autorepeat Pending State and an Autorepeating State.The gesture system keeps track of the most-recently reported userrequest, along with the finger identities that gestured that userrequest. The Autorepeat Pending State listens for “long down” fingerevents and waits to receive one “long down” for each finger identity ofthe preceding user request. The gesture detector terminates withmismatch if a “long down” is reported for a finger identity that was notpart of the previous user request. Upon detecting the required “longdowns,” the gesture system checks to see whether each finger has a touchcount of at least 2. If so, the gesture detector changes to theAutorepeating State.

Upon entering the Autorepeating State, the gesture detector starts atimer with a timeout period. If the maximum touch count was 2, thetimeout period is long (e.g. 333 ms), and if it is 3, the timeout periodis short (e.g. 200 ms), making for slow and fast autorepeating,respectively. On each timeout, the detector reports the previouslyreported user request to the host again and restarts the timer foranother timeout. The autorepeat detector terminates when at least one ofthe fingers lifts. It also terminates if one of the fingers beginssliding or if an additional finger touches for a long-down.

The gesture system can interpret any single-finger double- ortriple-touch or any two-finger double- or triple-touch as a request torepeat the previous user request, provided that it is followed by a“long down” period to signal a request to autorepeat, and provided thatthis behavior is compatible with the non-finger-mapped gesturessupported. This potentially generalizes autorepeating to supportnon-finger-mapped gestures as well.

8.2.3 Supported Non-Finger-Mapped Gestures

The cursoring and highlighting modes implement gestures without using afinger map. Their gestures are not technically finger-mapped gestures.The user may perform a calibration gesture while in the cursoring orhighlighting mode to exit the mode, produce a finger map, and entercharacter entry mode. While in character entry mode, the user maydouble-touch the thumb to induce an escape hold in the cursoring modeand triple-touch the thumb to induce an escape hold in the highlightingmode. Double- or triple-touching the thumb in cursoring mode establisheshighlighting mode.

The cursoring and highlighting modes provide the same gestures. Thesegestures use only a single finger and do not depend on the identity ofthe finger, except for the determination of highlighting mode. If afinger multi-taps but remains stationary while another gestures andcompletes, the first finger is deemed the thumb performing a hold toestablishing highlighting mode. This hold locks. The gesturing fingermoves in the FORWARD, BACKWARD, LEFT, or RIGHT ranges of directions toissue requests to move the cursor in these directions. The length of thegesture (that is, the gesture level) and the touch-count of the gesturetogether determine how far to move in that direction. The gestures movethe fingers by one character, one word, one line, one paragraph, onepage, or to the start or end of text. Moving the cursor in highlightingmode causes the characters between the cursor position at the start ofhighlighting and the current cursor position to be highlighted. This canbe accomplished by having each cursoring user request indicate whetherthe cursoring movement was done with highlighting. The host manages thehighlighting itself.

The gesture system also provides a set of gestures that are common toall three modes. In particular, they can be performed on top of a fingermap without being confused with finger-mapped gestures. The gestures aredesigned to be distinguishable from finger-mapped strokes andfinger-mapped adjacent two-fingered sweeps. The common gestures have twogesturing fingers coming together (“pinching”), going apart (“zooming”),spaced far apart (too far apart to be considered sweeps), or adjacentbut moving left or right, and they can be parameterized by touch count,direction, and gesture level. The common gestures represent userrequests such as cutting, copying, or pasting clipboard text, or issuingnew lines, line breaks, tabs, or back tabs. The common gestures alsorepresent requests to open a virtual keyboard or a table of symbolcharacters. A gesture can even terminate the gesture system, closing thesurface allocation.

With a suitable touch count to designate the meaning of a pinch or zoomgesture, pinching and zooming can be used to dynamically adjust thegesture level threshold distances. For example, suppose the gesturesystem supports separate gesture threshold distances for determininggesture level according to the direction of the gesture. A triple-touchpinch or zoom oriented left and right can then change the thresholddistances for determining gesture levels for left/right gestures inproportion to the change in distance between the fingers during thegesture. Likewise, a triple-touch pinch or zoom oriented forward andbackward can also change the threshold distances for determining gesturelevels for forward/backwards gestures in proportion to the change indistance between the fingers during the gesture.

The gesture system can also support a “scrubbing” gesture by limitingthe range of direction angles recognized as tickling. This allowsfingers that gesture at strong angles to the left or right to beinterpreted as scrubbing gestures if they move left and right at a rapidpace. A gesture of at least three extent vectors is sufficient.Scrubbing can be single-fingered or multi-fingered. Scrubbing gesturesare useful for representing user requests to undo and redo previous userrequests, because the scrubbing gesture motion is similar to the motionof erasing with a pencil's eraser.

8.3 Architectural Overview

The sample implementation includes the following architecturalcomponents:

-   -   1) Session Manager—The session manager is a singleton object        that manages the gesture system for the life of a gesture        session. It receives input events from the host for the surface        allocation, creates finger trackers as necessary to handle them,        and forwards the events to the appropriate finger trackers. The        session manager also manages the session configuration and the        active hand calibration.    -   2) Finger Tracker—A finger tracker (also called a “tracker”)        manages a particular finger for the full time that the finger        participates in a gesture. It receives raw input events,        accumulates them and translates them into the simpler finger        events that gesture detectors understand, sending finger events        to the gesture detectors via the session engine. Each finger        tracker also maintains finger-specific state, such as touch        counts, the initial touch location, and extent vectors        accumulated during a tickling calibration.    -   3) Slide Detector—A slide detector is an object that a finger        tracker employs to monitor a finger's path for the initial        slide. The finger tracker hands it each newly received point on        a finger's path. The slide detector collects the points,        checking them for the initial slide condition and informs the        finger tracker whether the new point resulted in an initial        slide. Once an initial slide occurs, the finger tracker        considers the finger to be thereafter sliding, so the finger        tracker discards the slide detector.    -   4) Session Engine—The session engine is a singleton state        machine that receives events from the finger trackers, manages        the life cycle of the gesture detectors, and forwards finger        events to the gesture detectors. It is responsible for waiting        for unrecognized gestures to complete before accepting new        gestures.    -   5) Gesture Detector—A gesture detector monitors finger events        for specific gestures. Each kind of gesture detector recognizes        a specific set of gestures. Each gesture detector receives        events from the session engine, attempts to find a matching        gesture, and reports the results of the attempt back to the        session engine. Upon finding a match, the gesture detector also        identifies a user request for the gesture and reports the user        request to the host when the gesture completes. This        specification provides details for the stroke gesture detector,        the sweep gesture detector, and the calibration gesture        detector. The sample implementation includes other gesture        detectors as well, such as a gesture detector for receiving        autorepeat gestures, pinch and zoom gestures, and non-adjacent        two-fingered gestures.    -   6) Hand Calibration—A hand calibration provides a channel-based        characterization of a particular user's hand. The sample        implementation maintains one hand calibration for the left hand        and another for the right hand, employing the appropriate one        for the hand in use. The hand calibration itself consists of        four finger calibrations, some of which can be inferred from        other finger calibrations during a calibration gesture. The        drumming gesture switches between the left and right hand        calibrations. The gesture system always draws the finger map for        the current hand calibration.

8.4 Input Events

The sample implementation assumes that the host emits the followinginput events:

-   -   1) Finger Down—The user has just placed a finger on the input        area. This event includes the coordinates of the location that        the finger touched. “Finger down” must be the first event issued        for a newly touching finger; it is issued for a finger prior to        issuing any “finger moved” or “finger up” event for the finger.    -   2) Finger Moved—A finger currently on the input area has moved        to a location different from its last reported location. This        event includes the coordinates of the location to which the        finger moved. It is up to the host whether to inform the gesture        system of movements of a finger that leaves the input area. This        event can only be issued for a finger that was previously        reported via “finger down.”    -   3) Finger Up—A finger that was previously placed on the input        area has just lifted from the touchscreen. This event need not        report coordinates. The event can only be issued for a finger        previously reported via “finger down,” but it must be reported        for a finger regardless of where on the touchscreen the finger        lifts.

Each input event also includes a host-assigned finger ID that enablesthe gesture system to associate multiple events with the same finger.This host finger ID is provided with the “finger down” event, remainsconstant across “finger moved” events, and is reported one last time at“finger up.” “Finger up” releases the ID for possible reuse by the hostin a subsequent “finger down.” Note that a “finger ID” is different froma “finger identity.” Finger IDs distinguish multiple touching fingerswithout regard to the identities that the fingers have on a hand. Eachinput event also includes a timestamp providing the time at which theevent occurred. In the case of “finger moved,” this is the time at whichthe finger arrived at the reported new location.

8.5 Session Manager

The session manager implements a gesture session. The host creates anddestroys session managers as needed. The host may specify handcalibrations for the session manager upon creation, but it need not,because the user can input hand calibrations via the session manager.Other components of the sample implementation look to the sessionmanager for the gesture system's configuration and active handcalibration.

8.5.1 Session Manager Configuration

The session manager keeps a hand calibration for both the left and righthands. The hand calibrations can be pre-configured when the sessionmanager is created.

8.5.2 Session Manager State Values

The session manager keeps a list of the active finger trackers. Thisincludes trackers for fingers that have recently lifted from thetouchscreen if the system deems that those fingers may yet touch theinput area again to extend their contributions to the gesture. The listalso includes fingers that have recently lifted for their residual holdperiods. The session manager also keeps track of which fingers are knownto be performing holds.

Because this gesture system supports non-finger-mapping gestures, thesession manager also maintains a session mode, indicating which class ofgestures is in effect. Two modes are supported: character entry mode andcursoring mode, the second of which disables the finger map and enablesgestures for moving the cursor. Cursoring mode effectively has twosubmodes: highlighting and not highlighting, depending on how cursoringmode was activated.

The session manager also indicates which hand calibration is the activeone.

8.5.3 Session Manager Services

The session manager provides these services to other components of thesystem.

8.5.3.1 Process an Input Event

The session manager receives input events from the host, processing oneevent at a time. The inputs events are “finger down,” “finger moved,”and “finger up.” Upon receipt of a “finger down” event, the sessionmanager performs the following steps:

-   -   1) Scan the active finger trackers for a finger tracker having        an ID equal to the host-provided finger ID. If a matching ID is        found, the host is reusing an ID that it had used with a        previous finger. In this case, re-assign the finger tracker to a        different ID not already in use by another finger tracker. This        conflict can arise because a finger tracker can remain active        representing a finger for a brief period after the finger lifts,        during residual holds.    -   2) Loop through the active finger trackers, asking each whether        the newly down finger is actually an additional touch for a        finger that lifted only moments ago. Each finger tracker bases        the determination on the new “finger down” coordinates and on        the input regions touched. If any finger tracker reports that        the new “finger down” is an additional touch for its associated        finger, reassign that finger tracker's ID to the ID that the        host provided for the new “finger down.” This resumes the finger        tracker using the new finger ID. The finger is presumably the        same finger that had previously produced the finger tracker. In        this case, send the new “finger down” event to the resumed        finger tracker and skip step (4).    -   3) Here the session manager has a newly participating finger. It        sends a “new finger elsewhere” event to each finger tracker.    -   4) Finally, the session manager creates a new finger tracker for        the newly down finger, initializing it with the provided host        finger ID, event time, event coordinates, and input region        touched.

For the “finger moved” and “finger up” events, the session manager justfinds the active finger tracker having an ID that matches the providedhost ID and sends that finger tracker the “finger moved” or “finger up”event.

8.5.3.2 Confirm Potential Hold Fingers

The session engine calls this service to have the session manager informthe finger trackers that any finger that was potentially serving as ahold was actually serving as a hold. The session manager loops throughall of the active finger trackers and sends a “confirm hold” event toeach. Each finger tracker's internal state indicates whether thetracker's associated finger was potentially performing a hold, allowingthe finger trackers to handle “confirm hold” events appropriately.

8.5.3.3 Set Active Hand Calibration

Changes the active hand calibration to the one provided.

8.5.3.4 Set Active Session Mode

This service asks the session manager to change the active session mode.Because mode-specific gesture detectors may be in effect, the sessionmode can only be changed when no fingers are touching the input area.The session manager enforces this.

8.6 Finger Trackers

A finger tracker monitors a finger that touches the input area. Ittranslates input events into finger events. Finger events are adistillation of input events into units that provide enough informationfor gesture detectors to make decisions. While finger events areintended for consumption by gesture detectors, they are sent to thegesture detectors via the session engine, which maintains the generalstate of the gesture. A finger tracker also represents a user's fingerand the properties accumulated to characterize the finger and thehistory of its participation in the current gesture. A finger trackerrepresents a finger not only during the time it is in contact with thetouchscreen, but also potentially for a brief time after it lifts fromthe touchscreen between touch counts or during a residual hold. Eachfinger tracker runs as an independent state machine, as depicted inFIGS. 62A-62B.

8.6.1 Finger Tracker Configuration

Each finger tracker has an ID that uniquely identifies the finger. Thesession manager may reassign the ID to resolve conflicts among fingerIDs. The finger tracker is also configured with the finger's initiallocation and time.

8.6.2 Finger Tracker State Values

The finger tracker implements a state machine and so maintains a statename. It also keeps track of the finger's current location and touchcount. It delegates responsibility to a slide detector for keeping trackof points on the finger's path to detect the initial slide. It alsokeeps track of whether the finger is a thumb that is designated as ananchor.

The finger tracker also calculates each extent vector for the finger andmakes the extent vectors available to the gesture detectors. The fingertracker itself records the first and second extent vectors for use indetermining the gesture level and the backbrush level.

8.6.3 Finger Tracker Services

A finger tracker provides these services to other components of thesystem.

8.6.3.1 Process an Input Event

Once the session manager has identified or created the appropriatefinger tracker to process an input event, it sends the event to thefinger tracker for processing. The finger tracker processes each inputevent according to the current state of the state machine.

8.6.3.2 Determine Whether Finger has Additionally Touched

This service determines whether a “new finger” event represents anadditional touch for the finger. The service reports a finger to haveadditionally touched if the finger tracker is in the “Possible Intertap”superstate and the finger either touches the input region that thefinger touched on its first touch or it touches the input area within acertain radius of the finger's most recent touch location. The radius isuser-configurable.

8.6.3.3 Process a “Confirm Hold” Event

A “confirm hold” event informs the finger tracker that if it is in astate that allows the finger to potentially serve as a hold, the fingeris actually serving as a hold. The finger tracker passes this event toits state machine to lock the finger in the hold state.

8.6.3.4 Process a “New Finger Elsewhere” Event

A “new finger elsewhere” event informs the finger tracker that anotherfinger has newly touched the input area, a finger for which no fingertracker previously existed. The finger tracker passes this event to itsstate machine, which terminates the finger tracker if it was in a statewaiting for the finger to touch the input area again. If the user beginsa new gesture, the gesture that the current finger was contributing tomust be done.

8.6.4 Finger Events

The finger tracker's state machine translates input events into fingerevents that the gesture detectors understand. Each finger eventdescribes an action of the finger that the finger tracker monitors. Thestate machine sends finger events to the session engine to forward tothe gesture detectors. The finger events are as follows:

-   -   1) New Touch—Reports that the finger has touched the input area,        possibly again if the finger had previously lifted to        subsequently increase the touch count.    -   2) Long Down—Reports that the finger has been touching the input        area for an extended period of time without gesturing.    -   3) Slide—Reports that the finger has moved in a way determined        to be intentional. Upon performing an initial slide, the initial        slide is reported as a slide, and each subsequent movement of        the finger is reported as a slide.    -   4) End Touch—Reports that the finger has lifted from the input        area. This event does not preclude the finger from subsequently        reporting a New Touch.    -   5) End Hold—Reports that a finger is no longer capable of        serving as a hold. The finger has lifted, and the residual hold        period has expired.

8.6.5 Finger Tracker State Machine

FIGS. 62A-62B depict a state-transition diagram for the finger tracker'sstate machine. The transition from the initial state to the Finger Touchstate represents the session manager creating a finger tracker. Thediagram depicts this as a “new finger” event. The actions of thistransition initialize the state machine. The transitions to the finalstate represent inactivating or destroying the finger tracker.

8.6.5.1 Finger Tracker States

The state machine employs the following states:

-   -   Potential Hold or Slide State 1015—Superstate in which the        finger is on the touchscreen but hasn't yet begun gesturing.    -   Finger Touch State 1016—State in which the finger has recently        touched the input area. The finger tracker starts in this state        when the finger first begins to contribute to a gesture. This is        the only state from which the finger may lift to touch again in        order to increase the touch count. Lifting and touching again        within the time constraints returns the finger tracker back to        this state.    -   Tap Expiration State 1017—State in which the finger hasn't yet        begun gesturing but has been in contact with the touchscreen for        too long to be able to lift again to increase the touch count.        Gesture detectors are generally still waiting for this finger to        do something before identifying the gesture, but some gesture        detectors may act on the length of time that the finger has been        touching.    -   Known Hold State 1018—State in which the finger contributed to a        previously completed gesture either by serving as a hold for the        gesture or by serving as an anchor for the hand while the hand        performed the gesture. (The only anchor that the sample        implementation supports is the thumb anchor.) Once a finger        tracker is in this state, the associated finger is free to move        wildly about without breaking the hold or anchor.    -   Sliding State 1019—State in which the finger is considered to be        sliding. A finger begins sliding when it performs an initial        slide, moving a distance of at least the minimum initial slide        length in no more than the maximum initial slide duration. Once        a finger begins sliding it remains in the Sliding State 1019        until the end of the gesture.    -   Finger Lifted State 1020—Superstate in which the finger has        lifted from the touchscreen but may still contribute to the        gesture, either by touching again to increase the touch count or        by briefly providing a residual hold.    -   Possible Intertap State 1021—Superstate in which the finger has        lifted from the touchscreen and still has time to touch again to        increase the touch count.    -   Intertap Residual Hold State 1022—State in which the finger has        lifted from the touchscreen and still has time to touch again to        increase the touch count. A finger in this state is also capable        of serving as a residual hold.    -   Intertap Suspended Hold State 1023—State in which the finger has        lifted from the touchscreen and still has time to touch again to        increase the touch count but has been up too long to serve as a        residual hold. This state exists distinctly from the Intertap        Residual Hold State 1022 in order to allow the residual hold        time and maximum multi-tap up time to be independent of each        other.    -   Residual Hold State 1024—State in which the finger has lifted        from the touchscreen and still has time to serve as a residual        hold. The finger is not able to further increase the touch count        from this state.

8.6.5.2 Finger Tracker Transitions

The state machine transitions employ the following symbols:

Symbol Meaning L Current location of the finger L₀ Initial location ofthe finger C_(T) Current touch count S_(L) Minimum initial slide lengthS_(T) Maximum initial slide duration (milliseconds) T_(D) Multi-tap downtimer, which once started, times out when the finger has been on thetouchscreen too long to lift and touch again to increase the touchcount. The duration of this timer is called the “maximum multi-tap downtime.” T_(U) Multi-tap up timer, which once started, times out when thefinger has been off of the touchscreen too long to touch again toincrease the touch count. The duration of this timer is called the“maximum multi-tap up time.” T_(L) Long-down timer, which determineswhen the Long Down event should be reported. The duration of this timeris called the “minimum long-down time.” T_(R) Residual hold timer, whichonce started, times out after the finger has been lifted too long to beable to register as a hold. The duration of this timer is called the“residual hold time.”

The state machine employs the following transition text:

-   -   Transition 1025—“new finger/start T_(D); L₀=L; C_(T)=1; if        (T_(L)>T_(D)) {start T_(L)}; send(new touch)”    -   Transition 1026—“T_(D) T/O/if (T_(L)=T_(D)) {send(long down)}”    -   Transition 1027—“finger up/clear T_(D),T_(L); start T_(U),T_(R);        send(end touch)”    -   Transition 1028—“finger moved [t(S_(L))>S_(T)]”    -   Transition 1029—“T_(L) T/O/send(long down)”    -   Transition 1030—“finger moved [t(S_(L))≤S_(T)]/clear        T_(D),T_(L); beginSlide( ); send(slide, initial)”    -   Transition 1031—“confirm hold/clear T_(D),T_(L)”    -   Transition 1032—“finger up/start T_(R); clear T_(L); send(end        touch)”    -   Transition 1033—“confirm hold”    -   Transition 1034—“finger moved/continueSlide( ); send(slide)”    -   Transition 1035—“finger up/send(end touch)”    -   Transition 1036—“finger moved”    -   Transition 1037—“finger up/start T_(R)”    -   Transition 1038—“T_(R) T/O”    -   Transition 1039—“T_(U) T/O/send(end tap)”    -   Transition 1040—“finger down/clear T_(U),T_(R); L₀=L;        C_(T)=C_(T)+1; start T_(D); if (T_(L)>T_(D)) {start T_(L)};        send(new touch)”    -   Transition 1041—“(T_(R) T/O or new finger elsewhere)/clear        T_(R); send(end hold)”    -   Transition 1042—“T_(U) T/O/send(end tap); send(end hold)”    -   Transition 1043—“new finger elsewhere/clear T_(U),T_(R);        send(end tap); send(end hold)”    -   Transition 1044—“confirm hold”    -   Transition 1045—“new finger elsewhere”

8.6.6 Finger Tracker Functions

The finger tracker state machine uses the following support functions.

8.6.6.1 t(x)—Time Taken to Travel Distance x

This function returns the amount of time the finger took to travel toits current location from the most recent location on its path that isat least a distance of x from the current location. The state machineonly uses this function for the value S_(L), which is the minimuminitial slide length. (Including S_(L) in the transition text makes thenature of the t(x) function clearer.) If the finger has not yettravelled a distance of S_(L), t(S_(L)) evaluates to infinity forcomparison purposes. In this case, the implementation could have t(x)return the maximum value of an integer.

8.6.6.2 beginSlide( )—Initiate Sliding

This function initiates sliding behavior. In particular, it establishesan estimated angle for the first extent vector and a half-plane orientedpartitioning oriented to the estimated angle for monitoring forsubsequent reversals of direction.

8.6.6.3 continueSlide( )—Continue Sliding

This function monitors for reversals of direction and collects extentvectors.

8.6.6.4 Send(e) or Send(e, i)—Send Finger Event

The two forms of this function send the finger event e to the sessionengine. The form “send(e, i)” also reports that the event represents aninitial slide. The ‘i’ here is just a flag.

8.7 Session Engine

The session engine manages the gesture detectors. It creates a new setof gesture detectors to handle each new gesture and passes finger eventsto the gesture detectors until one of them recognizes the gesture. Whena gesture detector finishes receiving the gesture, or when no gesturedetectors recognize the gesture, the session engine deletes all of thegesture detectors and waits for the next gesture. The session engine isalso responsible for waiting for an unrecognized gesture to completebefore attempting to recognize a subsequent gesture.

8.7.1 Session Engine State Values

The session engine maintains an array of gesture detectors. It createsthe array anew for each new gesture, selecting the kinds of gesturedetectors according to the active session mode, and it removes gesturedetectors from the array as they report failure to recognize the gesturethat the session engine is currently in the process of receiving.

8.7.2 Session Engine State Machine

FIG. 63 depicts a state-transition diagram for the session engine statemachine. Although it is not indicated, the state machine begins in theAccepting Gesture State 1050 and persists for the life of the sessionmanager.

8.7.2.1 Session Engine States

The state machine employs the following states:

-   -   Accepting Gesture State 1050—State in which the gesture system        is accepting gesturing input from the user. The session engine        remains in this state waiting for new gestures to begin and        waiting for existing gestures to complete.    -   Cleaning Up Gesture State 1051—State in which the gesture system        is not accepting gesture input from the user. The session engine        enters this state when none of the gesture detectors recognizes        the gesture. It remains in this state until the user lifts all        fingers that were gesturing. Hold fingers may remain touching,        however.

8.7.2.2 Session Engine Conditionals

The state machine employs the following conditionals, which evaluate totheir indicated functions:

-   -   Conditional 1052—“feed( )”    -   Conditional 1053—“noDetectors( )”    -   Conditional 1054—“noNonHolds( )”    -   Conditional 1055—“noNonHolds( )”

8.7.2.3 Session Engine Transitions

The state machine transitions employ the symbols IC, M, C, and X definedin the section titled “State-Transition Diagrams.” It also employs thefollowing transition text:

-   -   Transition 1056—“new touch/if (noDetectors( )){activate( )}”    -   Transition 1057—“*”    -   Transition 1058—“C complete/clearDetectors( )”    -   Transition 1059—“IC incomplete or X mismatch”    -   Transition 1060—“true”    -   Transition 1061—“false”    -   Transition 1062—“false”    -   Transition 1063—“true/confirmHolds( )”    -   Transition 1064—“new touch”    -   Transition 1065—“end touch”    -   Transition 1066—“*”    -   Transition 1067—“true/clearDetectors( )”    -   Transition 1068—“false”    -   Transition 1069—“end hold [! noDetectors( )]”    -   Transition 1070—“end hold [noDetectors(]”    -   Transition 1071—“M match/keepMatch( )”

8.7.3 Session Engine Functions

The session engine state machine uses the following support functions.

8.7.3.1 Activate( )—Activate Gesture Detectors

This function creates a new array of gesture detectors, selecting thedetectors according to the session mode. The session mode that supportsfinger maps creates one instance of each of the following gesturedetectors: stroke detector, sweep detector, and calibration detector.Additional non-finger-mapping detectors are added to the array to detectclipboard operations, cursor movement requests, and general state changerequests, such as to change to cursoring mode or to close the sessionmanager.

8.7.3.2 clearDetectors( )—Terminate Gesture Detectors

This function terminates all active gesture detectors. This function isnever called in a context that terminates a gesture that a gesturedetector has only partially matched a gesture; it is only called whengesture detectors have either reported a match or all mismatched.

8.7.3.3 confirmHolds( )—Confirm all Potential Holds

This function asks the session manager to confirm all potential holdfingers.

8.7.3.4 Feed( )—Feed Finger Event to Gesture Detectors

This function iterates over the gesture detector array and sequentiallypasses to each gesture detector the finger event that was just sent tothe session engine. Each gesture detector immediately processes thefinger event and reports one of the following results: incomplete,match, complete, or mismatch. The function terminates each gesturedetector that reports a mismatch, removing it from the array. It stopspassing the finger event to gesture detectors the first time a gesturedetector reports a match or a completion. The order in which the gesturedetectors appear in the array is not significant for the gesturedetectors that support finger mapping, but it can be significant whengesture detectors are included for other types of gestures.

8.7.3.5 keepMatch( )—Keep Only the Matched Detector

This function removes all gesture detectors from the array except forthe one that just indicated a match.

8.7.3.6 noDetectors( )—Whether there are No Detectors

Returns true if the gesture detector array is empty and false otherwise.

8.7.3.7 noNonHolds( )—Whether at Most Holds Remain

Returns true if there are no fingers other than hold fingers in thesession manager's list of active fingers, and returns false otherwise.In other words, returns true if either there are no active fingers orall active fingers are hold fingers, and returns false otherwise. In thecontext of this function, a hold finger is a locked hold finger or afinger that was serving as a hold in the present gesture at itscompletion according to the completing gesture detector, such as afinger that was touching the input area at completion without everhaving performed an initial slide or a finger for which a residual holdwas in effect.

8.8 Stroke Detector

A stroke detector detects a finger-mapped stroke gesture. It supportsgesture direction and backbrushing. Only two backbrush levels areavailable. The gesture direction of a stroke evaluates to the discretedirection for the angle of the stroke's first extent vector as read froma quarter-plane partitioning oriented to the forward angle for thefinger identity determined for the finger.

8.8.1 Stroke Detector State Machine

FIG. 64 depicts a state-transition diagram for the stroke detector statemachine. The transition from the initial state 1010 represents thecreation of the state machine, and the transitions to the final state1011 represent its destruction.

8.8.1.1 Stroke Detector States

The state machine employs the following states:

-   -   Potential Stroke State 1075—Superstate representing any state of        the state machine. The transitions 1083-1085 on this state could        also have been drawn from the “any state” 1012.    -   Stroke State 1076—State in which the user is gesturing the first        extent vector of the stroke. The transition 1081 to the        Back-Brush State 1077 is more strict than necessary, as it could        simply be based on the finger tracker detecting a second extent        vector.    -   Back-Brush State 1077—State in which the user is gesturing the        second extent vector of the stroke and hence backbrushing. The        same-state transition 1082 for sliding is overly strict, as it        could simply preclude the detection of a third extent vector.        There is no additional need to constrain the direction of a        backbrush.

8.8.1.2 Stroke Detector Transitions

The state machine transitions employ the following symbols, in additionto the symbols #A, #B, IC, M, C, and X defined in the section titled“State-Transition Diagrams”:

Symbol Meaning D Discrete direction of the vector that starts at thestarting location of the finger's current extent vector and ends at thefinger's most-recent location, according to a quarter-plane partitioningoriented to a reference angle equal to the forward angle for thefinger's identity. D₀ Discrete direction of the gesture. A value of 0means that it has not been assigned yet. L_(D) Minimum directiondetermination length. M Distance measure for the finger's currentlocation. r₀ Indicated user request, including a record of whether thethumb was interpreted as an anchor hold at the time the user request wasdetermined.

The state machine employs the following transition text. The notations#A and #B are “finger labels,” as described in the section titled“State-Transition Diagrams.” Note that according to this transitiontext, the first finger to slide matches #A. A second finger slidingwould subsequently match #B and cause the state machine to terminatewith a mismatch.

-   -   Transition 1078—“/D₀=0”    -   Transition 1079—“slide #A [D₀=0 or (M≤L_(D) & D≠op(D₀))]/D₀=D;        return(IC)”    -   Transition 1080—“slide #A [M>L_(D) & D=D₀]/return(IC)”    -   Transition 1081—“slide #A [D₀≠0 & D=op(D₀)]/return(IC)”    -   Transition 1082—“slide #A [D=op(D₀)]/return(IC)”    -   Transition 1083—“end touch #A [sliding( ) & r( )≠0]/r₀=r( );        report (r₀); return (C)”    -   Transition 1084—“end touch #A [sliding( ) & r( )=0]/return(X)”    -   Transition 1085—“slide #A/return(X)”    -   Transition 1086—“*/return(IC)”    -   Transition 1087—“slide #B/return(X)”

8.8.2 Stroke Detector Functions

The stroke detector state machine uses the following support functions.

8.8.2.1 Op(Base)—Opposite Direction

Returns the discrete direction opposite the indicated base direction ina quarter-plane partitioning.

8.8.2.2 r( )—Retrieve User Request

Returns a user request associated with the combination given by theselected finger identity, the discrete direction Do, the gesture levelof the first extent vector, the shift hold, if any, and whether thegesture backbrushed.

8.8.2.3 Report(Request)—Report User Request to Host

This function reports the indicated user request to the host. Inaddition, if the thumb was touching the input area at the time therequest was determined, and if the thumb was ignored as an anchor ratherthan interpreted as a shift further selecting the user request, have thefinger tracker for the associated finger designate the finger as ananchor.

8.8.2.4 Return(Result)—Return Result of Match Attempt

Returns to the session engine's feed( ) function the result ofattempting to match a finger event to a gesture. The possible resultsare incomplete, match, complete, and mismatch.

8.8.2.5 Sliding( )—Finger is Sliding

Returns true if the finger is sliding as a result of having performed aninitial slide.

8.9 Sweep Detector

The sweep detector implements two-fingered sweeps in any of fourdirections. The two fingers must travel in directions not more than π/2radians apart. The gesture level of the gesture is the higher of thegesture levels indicated for each finger separately, and the fingers maynot differ by more than one gesture level. Only two backbrush levels aresupported, and only one finger need backbrush to indicate backbrushing.The gesture direction of the sweep is determined from an orientedpartitioning oriented to the average of the forward angles of the twofingers, sampled at an angle equal to the average of the angles of thefingers' first extent vectors

8.9.1 Sweep Detector State Machine

FIGS. 65A and 65B depict a state-transition diagram for the sweepdetector state machine. The transition from the initial state 1010represents the creation of the state machine, and the transitions to thefinal state 1011 represent its destruction.

8.9.1.1 Sweep Detector States

The state machine employs the following states:

-   -   Sweep Pending State 1090—State in which the gesture detector is        waiting for a second finger to contribute to the gesture.    -   Sweep State 1091—State in which two fingers are in the process        of tracing their first extent vectors.    -   Half Back-Brush State 1092—State in which one of the two fingers        has begun backbrushing. This state waits for the second finger        to begin backbrushing, but it does not require that the second        finger do so.    -   Full Back-Brush State 1093—State in which both fingers are        backbrushing.    -   Completing Sweep State 1094—State in which one of the two        fingers has lifted. This state waits for the second finger to        lift before reporting the user request.    -   Mismatch Pending State 1095—State in which the second finger did        not lift soon after the first finger, indicating that the user        may have been performing a gesture other than a sweep. The        gesture detector enters this state on a timeout. It is an        artifact of the sample implementation that this state is even        necessary, as the very next finger event terminates the state.        The diagram need not have included this state.

8.9.1.2 Sweep Detector Transitions

The state machine transitions employ the following symbols, in additionto the symbols #A, #B, #X, #Y, #n, IC, M, C, and X defined in thesection titled “State-Transition Diagrams”:

Symbol Meaning D Discrete direction of the vector that starts at thestarting location of the finger's current extent vector and ends at thefinger's most-recent location, according to a quarter-plane partitioningoriented to a reference angle equal to the forward angle for thefinger's identity. C_(TA) Current touch count for finger #A, which isdetermined by the finger tracker. C_(TB) Current touch count for finger#B, which is determined by the finger tracker. D_(A0) Discrete directiondetermined for #A, the first finger to slide. A value of 0 means that ithas not been assigned yet. D_(B0) Discrete direction determined for #B,the second finger to slide. A value of 0 means that it has not beenassigned yet. D_(n0) Discrete direction determined for the currentfinger, #A or #B. Equal to either D_(A0) or D_(B0). G_(FA) Directionangle of the first extent vector for finger #A, as so far determined.G_(FB) Direction angle of the first extent vector for finger #B, as sofar determined. G_(S) Maximum sweep angle difference − maximumdifference allowed in the directions of the fingers' first extentvectors, by class equivalence, enforcing lockstep gesturing, set at 90°.L_(D) Minimum direction determination length r₀ Indicated user request,including a record of whether the thumb was interpreted as an anchorhold at the time the user request was determined T_(C) Sweep completiontimer, limiting the amount of time the user has to lift the sweep fingerthat has not lifted yet. The period of time is called the “maximumcompletion time.”

The state machine employs the following transition text:

-   -   Transition 1096—“/D_(A0)=0; D_(B0)=0”    -   Transition 1097—“slide #A [D_(A0)=0 or (M≤L_(D) & D≠op        (D_(A0)))]/D_(A0)=D; return(IC)”    -   Transition 1098—“slide #A [M>L_(D) & D=D_(A0)]/return(IC)”    -   Transition 1099—“(slide #B) [|C_(TA)−C_(TB)|≤1 & validSweep(        )]/D_(B0)=D; return(IC)”    -   Transition 1100—“end touch #A [sliding( )]/return(X)”    -   Transition 1101—“(slide #A or #B) (#n) [M≤L_(D) & validSweep(        )]/D_(n0)=D; return(IC)”    -   Transition 1102—“(slide #A or #B) (#n) [M>L_(D) & D=D_(n0) &        validSweep( )]/return(IC)”    -   Transition 1103—“(slide #A or #B) (#n) [D=op(D_(n0))]/if        (#n=#A){#X=#B} else {#X=#A}; return(IC)”    -   Transition 1104—“slide #X (#n) [D=D_(n0) & validSweep(        )]/return(IC)”    -   Transition 1105—“(slide #X) (#n) [D=op(D_(n0))]/return(IC)”    -   Transition 1106—“(slide #A or #B) (#n) [D=op        (D_(n0))]/return(IC)”    -   Transition 1107—“(end touch #A or #B) (#n) [|G_(FA)−G_(FB)|≥        G_(S) & r( )=0]/if (#n=#A) {#X=#B} else {#X=#A}; r₀=r( ); start        T_(C); return(IC)”    -   Transition 1108—“(end touch #A or #B) [|G_(FA)−G_(FB)|>G_(S) or        r( )=0]/return(X)”    -   Transition 1109—“slide #X/return (IC)”    -   Transition 1110—“new touch/clear T_(C); return(X)”    -   Transition 1111—“end touch #X/clear T_(C); report(r₀);        return(C)”    -   Transition 1112—“slide #Y/clear T_(C); return(X)”    -   Transition 1113—“T_(C) T/O”    -   Transition 1114—“*/return(X)”    -   Transition 1115—“*/return (IC)”    -   Transition 1116—“slide/return(X)”

8.9.2 Sweep Detector Functions

The sweep detector state machine uses the following support functions.

8.9.2.1 Op(Base)—Opposite Direction

Returns the discrete direction opposite the indicated base direction ina quarter-plane partitioning.

8.9.2.2 r( )—Retrieve User Request

Returns a user request associated with the combination given by theselected finger identities, the determined discrete direction, thegesture level of the first extent vector, the shift hold, if any, andwhether at least one of the fingers backbrushed.

8.9.2.3 Report(Request)—Report User Request to Host

This function reports the indicated user request to the host. Inaddition, if the thumb was touching the input area at the time therequest was determined, and if the thumb was ignored as an anchor ratherthan interpreted as a shift further selecting the user request, have thefinger tracker for the associated finger designate the finger as ananchor.

8.9.2.4 Return(Result)—Return Result of Match Attempt

Returns to the session engine's feed( ) function the result ofattempting to match a finger event to a gesture. The possible resultsare incomplete, match, complete, and mismatch.

8.9.2.5 validSweep( )—Validate Sweep Direction

This function performs input region conflict resolution, computes thecurrent discrete direction for the sweep, and returns false if the sweepis not a lockstep gesture, returning true otherwise.

If both fingers are assigned to the same input region, the functionreassigns one of them to an adjacent region, reassigning itscorresponding finger identity as well. If they were assigned to anend-of-series region, one of the fingers is arbitrarily reassigned tothe only adjacent region and to its finger identity as well. Otherwise,the finger nearest to an adjacent region is reassigned to that adjacentregion and to its finger identity.

To compute the current discrete direction, the function establishes anoriented partitioning by orienting a quarter-plane partitioning to areference angle that is calculated as the average of the forward anglesassociated with each finger of the sweep. The current discrete directionis the discrete direction indicated on this oriented partitioning at asampling angle equal to the average of the angles of the both fingers'first extent vectors, as those angles are so far determined.

The function allows the current discrete direction to vary from call tocall of validSweep( ) until both fingers have each travelled a distancegreater than the minimum direction determination distance. After that,the function verifies that the current discrete direction is identicalto the discrete direction that the function previously determined forthe gesture. If the verification fails, the function returns false.

The function also returns false if the first extent vectors of the twofingers are ever found to differ by more than 90°.

In all other cases, the function returns true.

8.10 Calibration Detector

The calibration detector simultaneously monitors for drumming,repositioning, and tickling gestures. It detects tickling on any numberof regions simultaneously at any time, but once one finger has tickled,the gesture detector is thereafter only able to receive ticklinggestures. Before the user has performed any tickling, the user also hasthe option to perform a drumming gesture. After drumming, and whilekeeping the fingers touching, the user may move the fingers around toreposition, re-orient, and scale the active hand calibration. If allfour non-thumb fingers are down, the hand calibration and finger maponly update when the fingers become relatively stationary. The user mayaccidentally lift any number of the non-thumb fingers whilerepositioning, as long as at least one non-thumb finger remainstouching, allowing the user to subsequently touch the fingers again toresume repositioning. The user may also reposition with just the indexand pinky fingers, in which case the finger map updates in real time.

8.10.1 Default Hand Calibration

The calibration detector implements a default hand calibration. Thedefault hand calibration applies when the user performs a drumminggesture before having performed a tickling gesture. Under thiscondition, the gesture system has rest locations for the fingers butdoes not have forward angles for the fingers. If the finger map is notconfigured to parallelize input regions, the gesture system assumesdefault forward angles for each finger.

The default forward angles are estimates of the average natural forwardangles of fingers based on measurements of just a few people's hands.They likely require refinement by measuring the tendencies of many morepeople's hands. The defaults are as follows:

Finger Identity Forward Angle left pinky 100°  left ring 90° left middle85° left index 80° right index 100°  right middle 95° right ring 90°right pinky 80°

The default forward angles are relative to the angle of an orientationline. The angles given are the finger's actual forward angles for a handcalibration with a strictly horizontal orientation line, which has anangle of 0 radians. The actual forward angle given for a finger in ahand calibration is the class equivalent direction angle of the sum ofthe above angle and the orientation line angle.

The calibration gestures that the calibration detector recognizesprovide rest locations for all of the fingers, so there is no need toinfer rest locations. A gesture system that must infer rest locations,such as one that only inputs hand calibrations via hand selectionslides, can reasonably assume rest locations that are evenly spacedapart.

8.10.2 Calibration Detector State Values

The calibration detector keeps track of the fingers that touch thesurface allocation and associates information with them via “fingerrecords.” A “finger record” is a structure that indicates an identityfor a finger, a past recorded location for the finger, and an averagedvector for the finger. The finger identities are THUMB, INDEX, MIDDLE,RING, PINKY, and UNKNOWN. Vectors are “added” to the averaged vector.The line segment of an averaged vector is an averaging of all of thevectors ever added to it. One end of the line segment is the average ofall vector termini that have higher y-coordinate values, and the otherend of the line segment is the average of all vector termini that havelower y-coordinate values. The approach requires that the finger mapalways be oriented in a vertical direction on the surface allocation,but the averaging could also have grouped termini in relation to anestimated orientation line having any orientation.

The calibration detector maintains two lists of finger records. The“active records” list contains one finger record for each finger that iscurrently touching. This list may include a record for the thumb. Theother is a list of “archived records,” which contains records fornon-thumb fingers that input extent vectors by tickling and thensubsequently lifted.

Because the thumb may optionally touch to anchor the hand whilegesturing, the detector does not always know with certainty whether aparticular active record is for the index finger or the thumb. To helpdeal with this ambiguity, the calibration detector also maintains a“thumb-or-index-finger-known” boolean indicating whether an activerecord has been assigned to either the thumb or the index finger withcomplete confidence.

8.10.3 Calibration Detector State Machine

FIGS. 66A and 66B depict a state-transition diagram for the calibrationdetector state machine. The transition from the initial state 1010represents the creation of the state machine, and the transitions to thefinal state 1011 represent its destruction.

8.10.3.1 Calibration Detector States

The state machine employs the following states:

-   -   Calibration Pending State 1120—State in which the calibration        detector is waiting for either a drumming gesture or a tickling        gesture.    -   Positioning State 1121—State in which the user is actively        positioning the finger map, with the finger map periodically        updating. While in this state, either all four non-thumb fingers        are touching, or only the index finger and the pinky finger are        touching. The finger map updates continually when only the index        and pinky fingers are touching; otherwise it updates only when        the fingers pause the act of repositioning. This is a        consequence of simultaneously monitoring for tickling, which        allows the user to reposition a finger map and then adjust the        region shapes and forward angles before lifting the fingers.    -   Suspended Positioning State 1122—State in which the user has        lifted one or more fingers during the process of positioning the        finger map, as may happen accidentally during gesturing. The        state is able to detect and identify fingers that touch again to        resume positioning, which may enable the user to be completely        unaware that a finger necessary for positioning the finger map        was temporarily lifted.    -   Tickling State 1123—State in which the user has tickled with at        least one finger and in which the user may continue to tickle        fingers until either all non-thumb fingers are lifted or the        gesture detector cannot properly assign identities to the        fingers.

8.10.3.2 Calibration Detector Transitions

The state machine transitions employ the following symbols, in additionto the symbols IC, M, C, and X defined in the section titled“State-Transition Diagrams”:

Symbol Meaning C_(M) The number of fingers mapped to non-thumb fingers.This is the number of active records assigned to finger identities otherthan THUMB and UNKNOWN. Maximum value is 4. T_(P) Positioning timer,timing out after a number of milliseconds equal to the “positioningperiod.” While the user is positioning a hand calibration, thecalibration detector periodically checks finger positions toperiodically update the position of the hand calibration. This allowsthe host or the gesture system to periodically draw the new finger map,if it draws them, so the user can see the finger map moving. It alsoallows the user to begin tickling at any time after positioning usingthe new position without having to first lift the fingers. T_(T)Tickling report timer, timing out after a number of milliseconds equalto the “tickling report period.” This timer is analogous to T_(P),except that it is used to periodically update the hand calibration whilethe user is tickling. In addition to giving the user visual feedbackduring calibration, it gives the user the freedom to lift fingers,accidentally or on purpose, and place them again in the newly adjustedregions to continue tickling.

The state machine employs the following transition text:

-   -   Transition 1124—“new touch [firstTouch( ) & atMostThumb( )]/if        (anchored( )) {addAnchor( )}; addFromCal( ); return(IC)”    -   Transition 1125—“new touch [!firstTouch( ) & !drumming(        )]/return(IC)”    -   Transition 1126—“new touch [!firstTouch( ) & drumming( ) &        !badThumb( ) & firstRepos( )]/recordPos( ); start T_(P);        return(M)”    -   Transition 1127—“end touch/remove( ); return(IC)”    -   Transition 1128—“new touch [firstTouch( ) & !atMostThumb(        )]/return(X)”    -   Transition 1129—“new touch [!firstTouch( ) & drumming( ) &        (badThumb( ) or !firstRepos( ))]/return(X)”    -   Transition 1130—“slide [checkTickling( ) & !ticklable(        )]/return(X)”    -   Transition 1131—“new touch [C_(M)<4 & reposition( ) & posAssign(        )]/clear T_(P); return(IC)”    -   Transition 1132—“new touch [C_(M)=4 & posAssign( )]/return(IC)”    -   Transition 1133—“T_(P) T/O/if (C_(M)=2 or stationary( ))        {reposition( )}; recordPos( ); start T_(P)”    -   Transition 1134—“end touch [!mapped( )]/remove( ); return(C)”    -   Transition 1135—“end touch [mapped( ) & reposition( )]/remove(        ); return(IC)”    -   Transition 1136—“new touch [(C_(M)=4 or (C_(M)<4 & reposition(        ))) & !posAssign( )]/clear T_(P); reportAnchor( ); return(C)”    -   Transition 1137—“end touch [C_(M)>1 & remove( ) & !positionable(        )]/return(IC)”    -   Transition 1138—“end touch [(C_(M)>1 & remove( ) & positionable(        )]/recordPos( ); reposition( )) start T_(P); return(IC)”    -   Transition 1139—“new touch [posAssign( ) & positionable(        )]/recordPos( ); reposition( ); start T_(P); return(IC)”    -   Transition 1140—“new touch [posAssign( ) & !positionable(        )]/return(IC)”    -   Transition 1141—“slide [checkTickling( ) & !ticklable( )]/clear        T_(P); if (C_(M)=2 or stationary( )) {reposition( )};        reportAnchor( ); return(C)”    -   Transition 1142—“new touch [!posAssign( )]/reportAnchor( );        return(C)”    -   Transition 1143—“end touch [C_(M)=1 & mapped( )]/reportAnchor(        ); return(C)”    -   Transition 1144—“slide [checkTickling( ) & !ticklable(        )]/reportAnchor( ); return(C)”    -   Transition 1145—“slide [tickling( ) & !checkTickling(        )]/resetAvg( ); return(IC)”    -   Transition 1146—“new touch/addFromCal( ); return(IC)”    -   Transition 1147—“slide [tickling( )=checkTickling( ) or        !tickling( ) & checkTickling( ) & ticklable( )]/return(IC)”    -   Transition 1148—“T_(T) T/O/calibrate( ); start T_(T)”    -   Transition 1149—“end touch [!mapped( )]/remove( ); return(IC)”    -   Transition 1150—“end touch [mapped( ) & C_(M)>1]/archiveAvg( );        remove( ); return(IC)”    -   Transition 1151—“slide [!tickling( ) & checkTickling( ) &        !ticklable( )]/clear T_(T); calibrate( ); reportAnchor( );        return(C)”    -   Transition 1152—“end touch [mapped( ) & C_(M)=1]/clear T_(T);        calibrate( ); reportAnchor( ); return(C)”    -   Transition 1153—“*/return(IC)”    -   Transition 1154—“slide [checkTickling( ) & ticklable( )]/clear        T_(P); start T_(T); return({right arrow over (M)})”

The transition text includes the following “macro” functions, whichabbreviate the text:

-   -   positionable( ): “mapped( ) & (C_(M)=4 or (C_(M)=2 & haveIndex(        ) & havePinky( )))”    -   ticklable( ): “!unknown( ) & (!thumb( ) or swap( ))”

8.10.4 Calibration Detector Functions

The calibration detector state machine uses the following supportfunctions:

8.10.4.1 addAnchor( )—Record a Thumb Anchor

This function puts a thumb record in the list of active records,designating the finger unchangeably as a thumb. It also sets thethumb-or-index-finger-known boolean to true.

8.10.4.2 addFromCal( )—Record Finger from Finger Map if Possible

If there is a finger map on the surface allocation, this functionidentifies the gesturing finger according to its originating region. Ifthe finger's initial touch location is not on an input region, or if nofinger map has been established, the finger identity is assigned thename UNKNOWN. The function then puts a record of this finger identity inthe list of active records.

8.10.4.3 Anchored( )—Designated as an Anchor

Returns true if the current finger is designated as an anchor, accordingto the finger tracker, and returns false otherwise.

8.10.4.4 archiveAvg( )—Archive Tickling Average

Puts the active record for the present finger in the list of archivedrecords. This makes the averaged tickling vectors available forincorporation into the hand calibration at the next tickling reporttimeout, even though the finger is no longer touching.

8.10.4.5 atMostThumb( )—No Holds or Just Thumb Hold

Returns true if there are no fingers performing holds or if the onlyhold is the thumb anchor, otherwise returns false.

8.10.4.6 badThumb( )—Bad Thumb Assignment

Returns true if any of the active records was previously designatedunchangeably a thumb and yet is no longer assigned to a thumb identity.Otherwise it returns false.

8.10.4.7 Calibrate( )—Revise Hand Calibration from Tickling

This function adjusts the active hand calibration with the fingers thatthe user has tickled during the current gesture. It first collectstogether the most recent averaged vectors available for each finger. Ifa finger has an active record, the averaged vectors are taken fromthere, but if not, the function uses averaged vectors from the archivedrecords. The function uses the line segments of these averaged vectorsas channel line segments for new finger calibrations. When averagedvectors are not available for a particular finger, the finger's previousfinger calibration is used instead.

After computing the new hand calibration, the function makes it active,producing an associated finger map, and deletes all the archivedrecords.

8.10.4.8 checkTickling( )—Check Tickling Status

This function returns true if it considers the finger to be tickling andfalse otherwise. If this function has already been called for thecurrent finger event, it just returns the current tickling status. Thefunction also just returns the finger's current tickling status if thefinger tracker has not collected another extent vector for the fingersince the last call to this function. Otherwise the function reevaluatesthe finger's tickling status.

Before reevaluating, the function appends the newest extent vector tothe list of extent vectors that it is monitoring, dropping the oldestextent vector in the list if necessary to keep the list at a maximumnumber of vectors equal to the minimum tickling vector count.

If the finger was already tickling but the monitored extent vectorsrepresent a span of more than a certain amount of time, tickling ceasesand all but the most recent vector is removed from the list. This amountof time is reasonably twice the maximum tickling start time. If thefinger was already tickling and remained tickling fast enough, thenewest vector is added to the finger's averaged vector.

If the finger wasn't already tickling and if the number of monitoredvectors equals the minimum tickling vector count and if the monitoredvectors represent a span of less time than the maximum tickling starttime, indicate the start of tickling, including all but the firstmonitored vector in the finger's averaged vector. The first vector isleft out because it may not be representative of the intended ticklingrange.

8.10.4.9 Drumming( )—Test for Drumming Gesture

This function returns true if it is able to recognize the gesture as adrumming gesture and false otherwise. The function makes thedetermination only once for a given finger event and returns thatprevious determination on subsequent calls to the function for the samefinger event. (This result caching is an artifact of thestate-transition diagram, allowing the diagram to call the function formultiple transition tests.)

When determining whether the gesture is a drumming gesture, the functionfirst adds an active record for the latest finger via a call toaddFromCal( ). After that, if the number of active records is neither 4nor 5, the function aborts returning false.

Otherwise the function sets the active records to the result ofgetSortedRecords( ) and identifies the last (most recent) finger amongthe active records to touch the touchscreen. The relation of this fingerto the fingers of the other active records determines whether thegesture is as drumming gesture and which hand—left or right—is beingindicated.

If the finger for the fourth active record is the last (most recent) totouch, the gesture indicates a left hand with the first active recorddesignating the pinky. If the finger for the first active record is thelast to touch and there are four active records, the gesture indicates aright hand with the first active record designating the index finger. Ifthe finger for the second active record is the last to touch and thereare five active records, the gesture indicates a right hand with thefirst active record designating the thumb. Otherwise the function abortsreturning false.

At this point the gesture is known to be a drumming gesture, the left orright handedness of the gesture is known, and the finger identity of thefirst active record is known, making it possible to deduce the fingeridentities of all the active records. The function does so and assignsthese finger identities to the active records.

Finally, the function sets the thumb-or-index-finger-known boolean totrue and returns a result of true.

8.10.4.10 firstRepos( )—Attempt to reposition a hand calibration

This function returns true if it is able to successfully create a fingermap from the locations of the index and pinky fingers and falseotherwise. The function makes the determination only once for a givenfinger event and returns that previous determination on subsequent callsto the function for the same finger event. (This result caching is anartifact of the state-transition diagram, allowing the diagram to callthe function for multiple transition tests.) Notice that this functionis only ever called when finger identities have been established for allfour non-thumb fingers and the left- or right-handedness of the gestureis known.

To generate a finger map from the index and pinky finger locations, thefunction first creates a new hand calibration. If a hand calibration isavailable for the particular hand that the user is using, the new handcalibration is based on the old hand calibration by scaling andreorienting the old one so that the new rest locations of the index andpinky fingers correspond to the current locations of the index and pinkyfingers. If no base hand calibration is available, the functionestablishes a new hand calibration with index and pinky finger restlocations at the current index and pinky finger locations, either byevenly distributing the rest locations for the intervening fingers or byusing the current locations of the intervening fingers as restlocations.

The system may opt to scale the gesture level ranges according to thehand breadth of the hand calibration. This is a way to automaticallyaccommodate the smaller range of motion of fingers on smaller hands. Onsystems that do this, it is possible for the user to select a handcalibration having gesture level 2 at a shorter distance than theminimum initial slide distance, in which case it becomes impossible fora finger to select gesture level 1. Conflicts may also arise amongvalues configured specifically for the user. When such conflicts arise,the function fails to produce a new hand calibration, leaves anypreceding hand calibration in place, and returns false. Otherwise, thefunction reassigns the hand calibration for the indicated hand, makesthe new hand calibration the active hand calibration, includinggenerating a finger map for it, and returns true.

8.10.4.11 firstTouch( )—First Touch of Calibration Gesture

Returns true if there are no active records yet and false otherwise.

8.10.4.12 getSortedRecords( )—Return Sorted Active Records

This function returns a list containing the active records sorted fromleft to right on the surface allocation, leftmost first in the list,according to the current positions of the fingers that correspond to therecords. To create the sorted list, the function calculates thedistances between every two pairs of fingers in the active records,associating both fingers with each distance (via their active records).The two fingers associated with the greatest of these distances arecompared (via their active records), and the one with the lowerx-coordinate value is selected as the leftmost finger. All distancesthat are not relative to this leftmost finger are then discarded. Theremaining distances are then sorted lowest-distance first. Finally, thefunction creates a new list, first adding the active record for theleftmost finger to the list and then adding the active records for eachremaining finger to the list according to the order of their associateddistances in the sort of distances, starting with the first distance inthe sort.

8.10.4.13 haveIndex( )—have an Index Finger

Returns true if an active record is assigned to an index finger identityand false otherwise.

8.10.4.14 havePinky( )—have a Pinky Finger

Returns true if an active record is assigned to an pinky finger identityand false otherwise.

8.10.4.15 Mapped( )—Mapped to a Non-Thumb Finger

Returns true if the active record for the current finger indicates afinger identity other than THUMB and UNKNOWN, and returns falseotherwise.

8.10.4.16 posAssign( )—Attempt to Assign Missing Identities

This function attempts to assign finger identities to active records notyet having known identities. It returns false if unusual circumstancesare encountered during the attempt and true otherwise. The functionmakes the determination only once for a given finger event and returnsthat previous determination on subsequent calls to the function for thesame finger event. (This result caching is an artifact of thestate-transition diagram, allowing the diagram to call the function formultiple transition tests.)

When attempting to determine the finger identities, this function firstputs a new finger record with unknown identity into the list of activerecords. The function returns false if the resulting number of activerecords is 6 or greater.

This function is only called when the left- or right-handedness of thegesture is known and at least one active record is assigned the identityof a non-thumb finger. If some active records have unknown identities,it is possible to deduce known identities for them in certaincircumstances. The process first entails calling getSortedRecords( ) toget a list of the active records sorted from left to right. Thecircumstances are as follows:

-   -   1) Five fingers are touching. In this case there is only one        valid assignment for the fingers, and this assignment includes        the thumb at one end.    -   2) The number of fingers touching on the pinky side of an active        record with known identity is equal to the number of fingers        that a hand actually has on the pinky side of the finger with        that identity. In this case, all of the active records on the        pinky side can be assigned finger identities. No equivalent        circumstance exists for the thumb side of the hand because the        thumb is optional when performing calibration gestures.    -   3) Finger identities are given for two active records, a series        of one or more active records with unknown identities spans        between them, and the number of active records in the series        equals the number of fingers actually on a hand between fingers        of the known two finger identities. In this case, all of the        active records between the active records with known identities        can be assigned finger identities.

This function determines all of the circumstances that apply and assignfinger identities to the active records accordingly. It is possible thatthe finger identities given for some active records do not conform toany configuration of a hand with the indicated left- orright-handedness. This function may optionally return false in thesesituations. Otherwise the function returns true, regardless of whetherany assignments were made.

The purpose of this function is to allow the user to accidentally liftfingers while moving them to reposition a finger map on the surfaceallocation. This way a user who accidentally lifts a finger does nothave to lift all of the fingers to terminate the gesture and startagain.

8.10.4.17 Positionable( )—Whether Finger Map is Positionable

Returns true if active records have been assigned to all four non-thumbfingers or if the only non-thumb fingers assigned to active records areboth the index finger and the pinky finger; otherwise returns false.

8.10.4.18 recordPos( )—Record Current Positions

This function iterates through all the active records and sets the pastrecorded location of each active record to the current position of thefinger associated with the record.

8.10.4.19 Remove( )—Remove Current Record

If this function has not yet been called for the current finger event,it removes the active record for the current finger from the list ofactive records and assigns the thumb-or-index-finger-known boolean tofalse if no active record is subsequently assigned to either the thumbor the index finger. Regardless, it always returns true, which allowsthe function to appear in conditional statements prior to subsequentconditional tests that require the active record to first have beenremoved.

8.10.4.20 reportAnchor( )—Report Anchor if Present

If there is an active record for the thumb, have the finger tracker forthe associated finger designate the finger as an anchor.

8.10.4.21 Reposition( )—Re-Attempt to Position a Hand Calibration

Calls firstRepos( ) but always returns true in order to allow it toappear in conditional statements prior to subsequent conditional teststhat must follow it. In the circumstances where this function is called,a hand calibration is always available, regardless of whether the callto firstRepos( ) succeeds.

8.10.4.22 resetAvg( )—Reset Averaged Vector

Clears the averaged vector for the current finger's active record sothat no previous vectors are included in subsequent line segmentdeterminations.

8.10.4.23 Return(Result)—Return Result of Match Attempt

Returns to the session engine's feed( ) function the result ofattempting to match a finger event to a gesture. The possible resultsare incomplete, match, complete, and mismatch.

8.10.4.24 Stationary( )—Whether the Fingers are Stationary

This function examines each active record to see whether the distancebetween the current location of the record's associated finger and thepast recorded location for the finger is greater than a maximumstationary distance. If any active record indicates a greater distance,this function returns false; otherwise it returns true.

8.10.4.25 Swap( )—Swap Thumb and Index Finger

This function returns true if it can swap the thumb and index fingeractive records, and it returns false otherwise. The function makes thedetermination only once for a given finger event and returns thatprevious determination on subsequent calls to the function for the samefinger event. (This result caching is an artifact of thestate-transition diagram, allowing the diagram to call the function formultiple transition tests.)

If the thumb-or-index-finger-known boolean is true, the function returnsfalse. Otherwise, the function looks for an active record for the indexfinger, and if it finds one, it reassigns it to the thumb. Regardless ofwhether it finds an existing index finger record, it reassigns thecurrent active record to an index finger, sets thethumb-or-index-finger-known boolean to true, and returns true.

8.10.4.26 Thumb( )—Finger is a Thumb

Returns true if the active record for the current finger is assigned athumb identity and returns false otherwise.

8.10.4.27 Unknown( )—Whether Finger is Undetermined

Returns true if the finger's active record indicates a finger type ofUNKNOWN and false otherwise.

8.11 Configuration Values

This specification has described numerous parameters for configuringgesture systems. Only rarely has it suggested values for thoseparameters. The specific values may depend on the characteristics of thetouch device, on the preferences of the user, or on both. It is alsopossible that some values can be universal for all devices and users.Except for the values set by the calibration gestures, the inventor hasnot determined which values can be made independent of either device oruser. Dependencies should become clearer as the gesture system isimplemented across devices and as users provide feedback. Users may bebetter served by restricting some values to a small set of possibilitiesthat users can select from, but these useful values have also not beendetermined. A gesture system may also implement calibration gesturesthat infer values instead of requiring that a user set them directly.This section presents values determined specifically for a Google Nexus7 tablet and specifically for the inventor.

8.11.1 Length Values

These configuration values report lengths and distances. These numbersare necessarily a function of the touch device, though some may also bea function of the user. The values given here are in pixels for a devicethat has about 85 pixels per centimeter.

Length Parameter Value Maximum tap radius - the maximum radius 40 aroundthe location of a previous touch in which a subsequent touch may bedetected to increase the touch count Maximum completion radius - themaximum 20 distance a gesturing finger may travel after the firstlifting of a finger of a selecting finger set; gestures exceeding thisdistance are unrecognized Maximum adjacent separation - the maximum 240distance between adjacent fingers, used to distinguish sweeps fromtwo-fingered gestures that are not finger-mapped Gesture level 1length - the minimum 9 distance for gesture level 1, which is also theminimum initial slide length Gesture level 2 length - minimum gesture 90level 2 distance, when not inferring from a calibrated hand breadthGesture level 3 length - minimum gesture 340 level 3 distance, when notinferring from a calibrated hand breadth Minimum slide leveldifference - when the 15 gesture system automatically adjusts gesturelevels according to the lengths of the average tickling vectors,deducing a differ- ence between successive slide levels that is lessthan this value produces an error Direction assignment distance - afinger 18 is assigned a discrete direction after traveling this distanceor greater Maximum stationary distance - maximum 16 distance that afinger may move while calibrating in order to be considered stationaryduring a single calibration positioning interval Minimum extent vectorlength - minimum 9 length of an extent vector

8.11.2 Fractional Extent Values

The gesture system uses fractional extent configuration parameters todetermine values as a function of values input via a calibrationgesture. Calibration gesture values are multiplied by these values todetermine the value that the gesture system uses.

Fractional Extent Parameter Value Maximum finger extent fraction -multiple 0.6 of the hand breadth that determines the distance forwardand backward from the orientation line that input regions extend, wheninput regions are not filling the surface allocation Gesture level 2breadth fraction - multiple 0.16 of the hand breadth yielding theminimum gesture level 2 length. Only applies when channel line segmentsare not available, such as from the hand calibration of a ticklinggesture.

8.11.3 Time Values

These parameter values specify time intervals and have units ofmilliseconds.

Time Parameter Value Maximum multi-tap down time - the maximum 300amount of time a finger has to lift again to increase the touch countMaximum multi-tap up time - the maximum 200 amount of time a finger canbe lifted before touching again to increase touch count Maximum initialslide time - the maximum 100 amount of time a finger has to slide thegesture level 1 length in order to perform an initial slide Minimum longdown time - the amount of 1000 time a finger must remain touchingwithout sliding to induce a “long down” Residual hold period - theamount of time 200 after a finger lifts during which it can beconsidered to be performing a residual hold Maximum completion time -the maximum 500 amount of time that the second finger of a sweep has tolift after the first finger lifts Slow autorepeat period - intervalbetween 333 reissuing user requests during a slow autorepeat Fastautorepeat period - interval between 200 reissuing user requests duringa fasts autorepeat Positioning period - interval at which a 150calibration gesture is tested for being stationary Tickling reportperiod - interval at which 1000 a calibration gesture updates the handcalibration and the finger map while tick- ling. 0 turns off thisdynamic updating Maximum start-of-tickling time - maximum 1250 amount oftime in which to input the minimum number of tickling vectors at theminimum rate in order to start tickling, derived from control values as1000 times the minimum tickling vector count divided by the minimumtickling vectors per second Minimum end-of-tickling time - minimum 2500amount of time for the minimum tickling vectors to be input in order toend tick- ling, derived as twice the maximum start- of-tickling time

8.11.4 Control Values

Some configuration parameters control the behavior of the gesturesystem. Those with boolean values are necessarily assigned eitheraccording to user preferences or according to the constraints of thedevice. For example, some touch devices may not be able to representfinger maps with arbitrary input regions in arbitrary orientations, andsome users may prefer to fix the forms of the input regions and theirorientations despite the abilities of the device.

Control Parameter Value Tickling vector window - the number of 5 mostrecent tickling vectors that the gesture system averages togetherBounded finger map - whether to bound true or false the finger mapforward and backward of a calibrated orientation line by a distancederived from the maximum finger extent fraction, thus freeing thesurface allocation outside these boundaries for other uses, such as foruse by the host Verticalize finger map - whether to orient true or falsea finger map so that the nearest average of the forward angles of thefingers is equal to the angle for the forward vertical direction, whichis the direction angle that points directly to the ‘top’ of the device.This option forces the hand to have a specific orientation with respectto the device. Parallelize finger map - whether to based true or false afinger map on a hand calibration that is derived from the actual handcalibration by setting all forward angles of the derived handcalibration to the nearest average of all forward angles of the actualcalibration. This option makes the input regions parallel “swim lanes.”When combined with verticalizing on a conven- tional tablet, the inputregions become a series of rectangles whose edges are parallel to thesides of the tablet. Center finger map horizontally - centers true orfalse the finger map horizontally on the surface allocation midwaybetween the ‘left’ and ‘right’ sides of the surface allocation Centerfinger map vertically - centers true or false the finger map verticallyon the surface allocation midway between the ‘top’ and ‘bottom’ sides ofthe surface allocation

8.12 State-Transition Diagrams

This specification includes state-transition diagrams for several statemachines. This section explains the notation that these diagrams use.The notation is modified from Unified Modeling Language (UML) forclarity and economy of expression. In order to satisfy draftingrequirements, text describing state transitions appears in thespecification rather than in the diagrams.

8.12.1 Graphical Elements

The state-transition diagrams include the following graphical elements:

-   -   1) States—Rounded rectangles represent states of the state        machine. The state machine is always in exactly one of the        states other than the “any state.” The large round black dots        also represent states, but these are states in which the state        machine is not running. The description of each state machine        summarizes each state.    -   2) Transitions—A transition is an arrow that originates at a        state and points to either the same state or to another state.        It represents an event that causes an action to occur or the        state of the state machine to change. Each transition is        associated with “transition text” that describes the event along        with actions performed. The transition text for each transition        appears in the specification rather than in the diagram.        Transition text has a particular format, as explained below.    -   3) Initial State—The initial state 1010 of a state machine is        given by a large black dot that is not enclosed in a circle or a        rounded rectangle. This represents the state of the state        machine not yet existing. A transition that originates at the        initial state 1010 indicates the creation of the state machine.        The transition from the initial state may optionally perform        actions, indicated by the portion of the transition text that        follows a slash. Although the initial state 1010 is identically        numbered in each state machine, each state machine comes into        existence independently on its own transition.    -   4) Running States—A rounded box not containing a black dot is a        “running state.” The specification names and describes each        running state.    -   5) Any State—The “any state” 1012 is the state given by two        concentric rounded boxes. In the state machine to which it        pertains, the “any state” 1012 provides default transitions for        each of the state machine's running states. When both the “any        state” 1012 and a running state provide a transition for the        same event, the running state transition occurs and the “any        state” transition does not. Although the “any state” is        identically numbered 1012 in each state machine, the “any state”        only applies to the state machine with which it is depicted.    -   6) Final State—The final state 1011 is given by a circle or        rounded rectangle containing a large black dot. It represents        the state of the state machine having been terminated. Some        state machines never terminate and run forever or until        destroyed by other means. These state machines do not have a        final state. Some diagrams depict more than one final state        1011, but all final state elements within a state machine are        equivalent.    -   7) Same-State Transitions—A transition that originates and ends        on the same state is a same-state transition. This transition        does not result in a state change. The function of the        transition is clear when it performs actions in response to an        event. However, a same-state transition need not perform an        action, indicating instead that the event is to be ignored and        neither to be interpreted as an error nor to be handled by an        “any state” transition.    -   8) Superstates—A superstate is region that encloses two or more        running states. The states contained within a superstate are the        “substates” of the superstate. A transition that originates on a        superstate indicates that the transition applies separately to        each of the superstate's substates. The diagram could instead        have been equivalently written with the superstate transition        duplicated for each of its substates. However, when superstate        and substate transitions conflict, the substate transition takes        precedence; a superstate transition only applies when no        substate transition applies for the same event. When a        transition originates and ends on the same superstate, the        transition is a same-state transition, not changing the state of        the state machine.    -   9) Conditionals—The session engine state machine in FIG. 63        employs a custom notation for expressing complex state        transitions. These are “conditionals,” illustrated in the figure        as diamonds 1052-1055. Each conditional corresponds to a        function that performs a test. Transition arrows can point to        conditionals, indicating that the conditional's test is to be        performed during the transition. The test is a function call.        This specification names the function associated with each        conditional. The function returns a result, which serves as an        event for transitions that originate on the conditional.        Transition arrows originating on the conditional point to states        or to other conditionals. The event indicated for a transition        originating on a conditional identifies the function result to        which the transition applies; when a conditional's function        yields the result given for a particular transition out of the        conditional, that particular transition occurs.

8.12.2 Transition Text

Each transition has “transition text” that describes the conditionsunder which the transition occurs and the actions that are performedwhen the transition occurs. Transition text has one of the following twogeneral formats:

-   -   event name(#n)[conditions]/actions    -   (event names)(#n)[conditions]/actions

The first segment in each format, providing one or more event names, isoptional on the transition from the initial state 1010 but mandatory inall other transitions. The remaining segments may or may not appear in atransition. The segments are as follows:

-   -   1) event name—Name of the event that causes the transition. An        event name of asterisk (‘*’) indicates that the event is matched        if no other transition of the originating state matches. When        the transition originates on a conditional, the event name        identifies a return value of the test function that the        conditional called.    -   2) (event names)—When the names of the events are listed in        parentheses, several different events are each capable of        causing the transition. The parentheses disambiguate the names        from the rest of the transition text. A single event may also be        represented in parentheses.    -   3) (#n)—When this segment occurs after an event name, #n becomes        assigned to the “finger label” for the current finger. For        example, if the transition text begins with “(slide #A or        #B)(#n)”, the first finger to slide is given the label #A and        the second finger to slide is given the label #B. Regardless of        which finger matches the event, #n indicates the finger label of        the finger, whether it was #A or #B. See the “Finger Labels”        notation, below.    -   4) [conditions]—Further constraints on when the transition        occurs are given as conditions in brackets. The ampersand (′&′)        represents a boolean AND operator. The word ‘or’ indicates an        inclusive boolean OR operator. The AND operator has higher        precedence than the OR operator, as is conventional. An        exclamation point (‘!’) indicates a logical NOT, negating the        truth value of the immediately following term, as is        conventional in C-like programming languages. Within a        conditional expression, the ‘=’ sign is an equality comparison,        not an assignment. Vertical bars designate the mathematical        absolute value operation on the value enclosed between the        vertical bars.    -   5) /actions—Transitions are capable of performing actions in        addition to changing the state of the state machine. Actions are        listed after the slash and include variable assignments and        function calls. Semi-colons delimit multiple sequential actions.        Transitions may perform actions even if they are directed to        conditionals. The ‘=’ is a comparison when it appears in “if”        statements and it is a variable assignment operation otherwise.        The conditionally executed statements of “if” and “else”        statements occur in curly braces (‘{’ and ‘}’) after the        associated “if” or “else” clause, as is conventional in C-like        programming languages.

8.12.3 Shorthand Notation

The diagrams express the transition text in a shorthand notation. Thenotation helps to improve clarity by helping to reduce clutter. Thenotation is as follows:

-   -   1) Finger Labels—A “finger label” is given as a pound sign (‘#’)        followed by a capital letter. Finger labels distinguishing        detected fingers. “#A” and “#B” respectively signify the first        finger and second fingers for which the state machine receives        events. The expression “if(#n=#A)” tests the variable #n to see        if it represents the finger with label #A. The notation “#X=#A”        assigns “#X” the finger label “#A”. After the “#X” assignment,        “#Y” will identify the next finger, analogous to “#B” being the        finger label that follows “#A”. Think of the finger labels as        binding to the fingers as they arrive.    -   2) Variables—Variables are given as capital letters with        optional subscripts. Most variables are local to the state        machine, so that multiple state machines may run concurrently        without affecting each other's variable. However, a variable        given in italics is shared among all instances of the same kind        of state machine. A variable having a subscript that begins with        a lower case ‘n’ represents another variable, with the value of        “#n” in the transition text selecting that variable, selecting        either subscript ‘A’ or ‘B’ for finger labels “#A” or “#B”.    -   3) Recorded Variables—A recorded variable is a variable that        records the current value of another variable for possible use        later after the other variable has changed value. A recorded        variable is given as the variable it records followed by the        subscript zero. (For example, L₀ is a recorded value of the        variable L.) If the variable being recorded already has a        subscript, a zero is appended to that subscript. The transition        text explicitly shows the assignments of recorded variables.    -   4) Functions—Functions are given as lower case letters followed        by zero or more parenthesized parameters. Functions always        include the parentheses, whether or not they have any        parameters. Functions may also access variables not provided to        them in parenthesis and not depicted on the state-transition        diagram. Functions may also change object state other than the        state of the state machine, even if called from within the        conditions of state transition text. Functions may or may not        return values for use in transition conditions and actions.    -   5) Enumerated Types—Enumerated types are given as capital        letters that are bold and underlined. In particular, the state        machines employ an enumerated type for the matching results of        gesture detectors.

Gesture Detector Matching Results IC INCOMPLETE M MATCH C COMPLETE BXMISMATCH

-   -   6) Timers and Timeouts—A timer is represented as a capital bold        italic T having a subscript. (For example, T_(D).) The subscript        is a letter that identifies the timer. State machines start and        clear timers. When a timer is started, a timeout will occur a        number of milliseconds later that depends on the timer started,        unless the timer is canceled first. When a timeout occurs, an        event is sent to the state machine to report the timeout. The        event name is given as the symbol for the timer followed by the        letters “T/O”. (For example, “T_(D) T/O”.) If the timer is        canceled before the timeout occurs, no timeout event is sent to        the state machine for the most recent request to start the        timer. Each instance of the state machine maintains its own        timers, so that now two state machines can start or cancel each        other's timers.

What is claimed is:
 1. A method of user input on a touch deviceassociated with a host that interprets user requests in response to userinput, said method comprising the steps of: detecting a first set of atleast three fingers touching a surface during a first gesture;subsequent to said step of detecting a first set, concluding processingof a second gesture on said surface; subsequent to said step ofconcluding processing of a second gesture, detecting a selecting fingerset of gesturing fingers on said surface; determining a selectedidentity set comprising a distinct finger identity for each finger ofsaid selecting finger set, with said selected identity set beingdependent on a property of said first gesture selected from the groupconsisting of the order in which fingers of said first set touched saidsurface and the direction of movement of at least one finger of saidfirst set; determining a user request dependent on said selectedidentity set and on the gestured path of at least one finger of saidselecting finger set; and reporting said user request to said host.
 2. Amethod according to claim 1, wherein said property of said first gestureis said order in which fingers of said first set touched said surface.3. A method according to claim 2, wherein said method includes the stepsof: (i) determining a series of fingers comprising fingers of said firstset ordered dependent on touch locations of said fingers of said firstset, with said series of fingers having at least three fingers andincluding two end-of-series fingers and an indicator finger, saidindicator finger being selected from the group consisting of the firstfinger of said series of fingers to touch, the second finger of saidseries of fingers to touch, the second-from-last finger of said seriesof fingers to touch, and the last finger of said series of fingers totouch; and (ii) identifying an indicated end-of-series finger as thefinger of said two end-of-series fingers nearest in said series offingers to said indicator finger; and wherein said selected identity setis dependent on said indicated end-of-series finger.
 4. A methodaccording to claim 3, wherein said method further includes the steps of:(iii) establishing regions on said surface such that the initial touchlocation of each finger of said series of fingers is in a distinctregion; and (iv) defining a correspondence of said regions with a seriesof finger identities dependent on said indicated end-of-series finger;and wherein said selected identity set is dependent on saidcorrespondence and on a touch location of each finger of said selectingfinger set.
 5. A method according to claim 4, wherein said user requestis dependent on the length of said gestured path.
 6. A method accordingto claim 5, wherein said series of fingers has four fingers.
 7. A methodaccording to claim 6, wherein said indicator finger is both said lastfinger of said series of fingers to touch and said indicatedend-of-series finger.
 8. A method according to claim 6, wherein saidstep of determining a series of fingers selects fingers from said firstset such that (a) the first and last fingers of said series of fingersto touch are on opposite halves of said series of fingers, and (b) atleast one finger of said series of fingers has an initial touch timethat is between the initial touch times of said two end-of-seriesfingers.
 9. A method according to claim 8, wherein said first setincludes a finger not in said series of fingers having an initial touchtime that precedes the initial touch time of the third finger of saidseries of fingers to touch said surface.
 10. A method according to claim1, wherein said property of said first gesture is said direction ofmovement of said at least one finger of said first set.
 11. A methodaccording to claim 10, wherein said method includes the steps of: (i)detecting that the fingers of said first set moved during said firstgesture; and (ii) determining a finger of said first set to be a thumb;wherein said at least one finger of said first set includes a non-thumbfinger that is not said thumb; and wherein said selected identity set isdependent on the hand-relative positioning of said thumb in said firstgesture.
 12. A method according to claim 11, wherein said user requestis dependent on the length of said gestured path.
 13. A method accordingto claim 12, wherein said method further includes the steps of: (iii)establishing regions on said surface such that a touch location of eachfinger of a second set of fingers is in a distinct region, with saidsecond set comprising each finger of said first set other than saidthumb; and (iv) defining a correspondence of said regions with a seriesof finger identities dependent on said hand-relative positioning of saidthumb and on the direction of movement of said non-thumb finger; andwherein said selected identity set is dependent on said correspondenceand on a touch location of each finger of said selecting finger set. 14.A method according to claim 13, wherein said method further establishesa range of directions that is dependent on said direction of movement ofsaid non-thumb finger; and wherein said step of determining a userrequest further determines said user request dependent on whether thedirection of said gestured path is in said range of directions.
 15. Amethod according to claim 12, wherein said method further establishes arange of directions that is dependent on the direction of movement ofsaid non-thumb finger; and wherein said step of determining a userrequest further determines said user request dependent on whether thedirection of said gestured path is in said range of directions.
 16. Asystem for user input on a touch device associated with a host thatinterprets user requests in response to user input, said systemcomprising: a calibrating system that detects a first set of at leastthree fingers touching a surface during a first gesture; a gesturesystem that: (i) subsequent to said step that detects a first set,concludes processing of a second gesture on said surface; (ii)subsequent to said step that concludes processing of a second gesture,detects a selecting finger set of gesturing fingers on said surface;(iii) determines a selected identity set comprising a distinct fingeridentity for each finger of said selecting finger set, with saidselected identity set being dependent on a property of said firstgesture selected from the group consisting of the order in which fingersof said first set touched said surface and the direction of movement ofat least one finger of said first set; (iv) determines a user requestdependent on said selected identity set and on the gestured path of atleast one finger of said selecting finger set; and (v) reports said userrequest to said host.
 17. A system according to claim 16, wherein saidproperty of said first gesture is said order in which fingers of saidfirst set touched said surface.
 18. A system according to claim 17,wherein said calibrating system: (i) determines a series of fingerscomprising fingers of said first set ordered dependent on touchlocations of said fingers of said first set, with said series of fingershaving at least three fingers and including two end-of-series fingersand an indicator finger, said indicator finger being selected from thegroup consisting of the first finger of said series of fingers to touch,the second finger of said series of fingers to touch, thesecond-from-last finger of said series of fingers to touch, and the lastfinger of said series of fingers to touch; and (ii) identifies anindicated end-of-series finger as the finger of said two end-of-seriesfingers nearest in said series of fingers to said indicator finger; andwherein said selected identity set is dependent on said indicatedend-of-series finger.
 19. A system according to claim 18, wherein saidcalibrating system further: (iii) establishes regions on said surfacesuch that the initial touch location of each finger of said series offingers is in a distinct region; and (iv) defines a correspondence ofsaid regions with a series of finger identities dependent on saidindicated end-of-series finger; and wherein said selected identity setis dependent on said correspondence and on a touch location of eachfinger of said selecting finger set.
 20. A system according to claim 19,wherein said user request is dependent on the length of said gesturedpath.
 21. A system according to claim 20, wherein said series of fingershas four fingers.
 22. A system according to claim 21, wherein saidindicator finger is both said last finger of said series of fingers totouch and said indicated end-of-series finger.
 23. A system according toclaim 21, wherein said step that determines a series of fingers selectsfingers from said first set such that (a) the first and last fingers ofsaid series of fingers to touch are on opposite halves of said series offingers, and (b) at least one finger of said series of fingers has aninitial touch time that is between the initial touch times of said twoend-of-series fingers.
 24. A system according to claim 23, wherein saidfirst set includes a finger not in said series of fingers having aninitial touch time that precedes the initial touch time of the thirdfinger of said series of fingers to touch said surface.
 25. A systemaccording to claim 16, wherein said property of said first gesture issaid direction of movement of said at least one finger of said firstset.
 26. A system according to claim 25: wherein said calibratingsystem: (i) detects that the fingers of said first set moved during saidfirst gesture; and (ii) determines a finger of said first set to be athumb; wherein said at least one finger of said first set includes anon-thumb finger that is not said thumb; and wherein said selectedidentity set is dependent on the hand-relative positioning of said thumbin said first gesture.
 27. A system according to claim 26, wherein saiduser request is dependent on the length of said gestured path.
 28. Asystem according to claim 27, wherein said calibrating system further:(iii) establishes regions on said surface such that a touch location ofeach finger of a second set of fingers is in a distinct region, withsaid second set comprising each finger of said first set other than saidthumb; and (iv) defines a correspondence of said regions with a seriesof finger identities dependent on said hand-relative positioning of saidthumb and on the direction of movement of said non-thumb finger; andwherein said selected identity set is dependent on said correspondenceand on a touch location of each finger of said selecting finger set. 29.A system according to claim 28, wherein said calibrating system furtherestablishes a range of directions that is dependent on said direction ofmovement of said non-thumb finger; and wherein said step that determinesa user request further determines said user request dependent on whetherthe direction of said gestured path is in said range of directions. 30.A system according to claim 27, wherein said calibrating system furtherestablishes a range of directions that is dependent on the direction ofmovement of said non-thumb finger; and wherein said step that determinesa user request further determines said user request dependent on whetherthe direction of said gestured path is in said range of directions. 31.A system according to claim 16, wherein said calibrating system is saidgesture system.
 32. A non-transitory computer-readable medium carryingthereon instructions which when executed by a processor perform a methodof user input on a touch device associated with a host that interpretsuser requests in response to user input, said method comprising thesteps of: detecting a first set of at least three fingers touching asurface during a first gesture; subsequent to said step of detecting afirst set, concluding processing of a second gesture on said surface;subsequent to said step of concluding processing of a second gesture,detecting a selecting finger set of gesturing fingers on said surface;determining a selected identity set comprising a distinct fingeridentity for each finger of said selecting finger set, with saidselected identity set being dependent on a property of said firstgesture selected from the group consisting of the order in which fingersof said first set touched said surface and the direction of movement ofat least one finger of said first set; determining a user requestdependent on said selected identity set and on the gestured path of atleast one finger of said selecting finger set; and reporting said userrequest to said host.
 33. A non-transitory computer-readable mediumaccording to claim 32, wherein said property of said first gesture issaid order in which fingers of said first set touched said surface. 34.A non-transitory computer-readable medium according to claim 33, whereinsaid method includes the steps of: (i) determining a series of fingerscomprising fingers of said first set ordered dependent on touchlocations of said fingers of said first set, with said series of fingershaving at least three fingers and including two end-of-series fingersand an indicator finger, said indicator finger being selected from thegroup consisting of the first finger of said series of fingers to touch,the second finger of said series of fingers to touch, thesecond-from-last finger of said series of fingers to touch, and the lastfinger of said series of fingers to touch; and (ii) identifying anindicated end-of-series finger as the finger of said two end-of-seriesfingers nearest in said series of fingers to said indicator finger; andwherein said selected identity set is dependent on said indicatedend-of-series finger.
 35. A non-transitory computer-readable mediumaccording to claim 34, wherein said method further includes the stepsof: (iii) establishing regions on said surface such that the initialtouch location of each finger of said series of fingers is in a distinctregion; and (iv) defining a correspondence of said regions with a seriesof finger identities dependent on said indicated end-of-series finger;and wherein said selected identity set is dependent on saidcorrespondence and on a touch location of each finger of said selectingfinger set.
 36. A non-transitory computer-readable medium according toclaim 35, wherein said user request is dependent on the length of saidgestured path.
 37. A non-transitory computer-readable medium accordingto claim 36, wherein said series of fingers has four fingers.
 38. Anon-transitory computer-readable medium according to claim 37, whereinsaid indicator finger is both said last finger of said series of fingersto touch and said indicated end-of-series finger.
 39. A non-transitorycomputer-readable medium according to claim 37, wherein said step ofdetermining a series of fingers selects fingers from said first set suchthat (a) the first and last fingers of said series of fingers to touchare on opposite halves of said series of fingers, and (b) at least onefinger of said series of fingers has an initial touch time that isbetween the initial touch times of said two end-of-series fingers.
 40. Anon-transitory computer-readable medium according to claim 39, whereinsaid first set includes a finger not in said series of fingers having aninitial touch time that precedes the initial touch time of the thirdfinger of said series of fingers to touch said surface.
 41. Anon-transitory computer-readable medium according to claim 32, whereinsaid property of said first gesture is said direction of movement ofsaid at least one finger of said first set.
 42. A non-transitorycomputer-readable medium according to claim 41, wherein said methodincludes the steps of: (i) detecting that the fingers of said first setmoved during said first gesture; and (ii) determining a finger of saidfirst set to be a thumb; wherein said at least one finger of said firstset includes a non-thumb finger that is not said thumb; and wherein saidselected identity set is dependent on the hand-relative positioning ofsaid thumb in said first gesture.
 43. A non-transitory computer-readablemedium according to claim 42, wherein said user request is dependent onthe length of said gestured path.
 44. A non-transitory computer-readablemedium according to claim 43, wherein said method further includes thesteps of: (iii) establishing regions on said surface such that a touchlocation of each finger of a second set of fingers is in a distinctregion, with said second set comprising each finger of said first setother than said thumb; and (iv) defining a correspondence of saidregions with a series of finger identities dependent on saidhand-relative positioning of said thumb and on the direction of movementof said non-thumb finger; and wherein said selected identity set isdependent on said correspondence and on a touch location of each fingerof said selecting finger set.
 45. A non-transitory computer-readablemedium according to claim 44, wherein said method further establishes arange of directions that is dependent on said direction of movement ofsaid non-thumb finger; and wherein said step of determining a userrequest further determines said user request dependent on whether thedirection of said gestured path is in said range of directions.
 46. Anon-transitory computer-readable medium according to claim 43, whereinsaid method further establishes a range of directions that is dependenton the direction of movement of said non-thumb finger; and wherein saidstep of determining a user request further determines said user requestdependent on whether the direction of said gestured path is in saidrange of directions.